In ODS-5 there is support for filenames which contain ISO Latin-1 or Unicode plane 0 characters. Filenames are flagged with a "Name type" which can be either ODS-2, ISL-1 or UCS-2. In the following these types are used to describe UTF-8 support.

ODS-2 is used for ODS-2 compatible filenames, with one exception or addition: there can be lowercase characters in the filename. (These are not legal characters in ODS-2, but it seems, if one could get these into a directory file and into the index file, ODS-2 would accept them.)

ISL-1 is used for filenames with
1) non-printable ASCII characters, 0x00 - 0x1f, 0x7f
2) ASCII characters - other than lowercase characters, which are not allowed in ODS-2 and
3) non-ASCII characters (8-bit), 0x80 - 0xff.

Ad 1) Non-printable ASCII characters in filenames need to be escaped for input and are escaped tor output. On the VMS command line, the escape sequence for them is ^xx, where xx is the (hexadecimal) code point in ASCII.
Ad 2) These are characters, which are not allowed in ODS-2, at all or have a specific meaning in the filename. '~', '#' and '{' are examples for the first category; '%', and space are examples for the first category and second category; and ':', '[' and '.' are examples of the second category. On the VMS command line, all characters, which have a special meaning, '%' is a wildcard, space is a command line delimiter and ':', '[', '.' are file specification delimiters, need to be escaped (for input) and are escaped (for output) when used in a filename. The escape symbol on VMS is ^, which is followed by the ASCII character - except for space, which for output is replaced by an underscore ('_').
Ad 3) On the VMS command line, these require an ISO Latin-1 character set to be loaded for the terminal for input and output of the filename. On the VMS command line, the escape sequence for for non-printable characters (0x80 - 0x9f, 0xa0) is ^xx, where xx is the (hexadecimal)code point in ISO Latin-1. For unknown reasons, 0xff is seen as non-printable as well, but it really is printable, it is 'ÿ'.

UCS-2 is used for filenames with non-ISO Latin-1 characters. That is, for Unicode plane 0 characters, that have a non-zero value in the upper byte of the UCS-2 encoding. On the VMS command line, when such characters are used in a filename, these characters need to be escaped (for input) and are escaped (for output). The escape sequence for them is "^Uxxxx", where xxxx is the code point in the Unicode.

On the VMS command line, when you enter a filename with Unicode characters as ^Uxxxx and with zeros in all the upper bytes, the ODS-5 file name type is ISL-1 or ODS-2, depending on the lower byte.

In the ods5 file system, without UTF-8 support, disabled with the mount option vtf7, ODS-2 and ISL-1 filenames are shown as-is. That is, when listing files from the command line in a terminal, non-printable ASCII characters may not show or may be interpreted (tab, line feed, etc.) and the terminal needs to have an ISO Latin-1 character set loaded for non-ASCII characters, otherwise the filenames are mis-interpreted, usually there is an '?' shown for the non-ASCII character. UCS-2 filenames are shown with escape sequences for non-ISO Latin-1 characters: "?xxxx", where xxxx is the code point in the Unicode. (The escape symbol '?' was choosen, because ODS-5 file names can not contain the wildcard characters '?' and '*'.)

For vtf7, printing of non-printable ASCII/ISL-1 may change in a future release. A "?xxxx" escape sequence seems to do a better job. However, as of today, such filenames are rare.

With UTF-8 support, enabled with the mount option utf8 or the absence of the mount option vtf7, the ods5 file system shows all filenames as UTF-encoded strings. Again, non-printable ASCII characters may not show or may be interpreted (tab, line feed, etc.) Usually they are shown as a question mark. For example 002-?.;1 which is a filename containing 0x02. In the bash, to enter such a filename when using filename completion, a ^B is shown, which is the associated control character for 0x02: 

# ls 002-^B.\;1
002-?.;1
#
Because the VMS/ODS file name spaces don't really map, there are exceptions.

On VMS "/" is a valid character filename as well as ^00. So you can have the filename "/" and one consisting of the NUL character, which one might describe as "\x00".

On VMS they are shown as "^/.;1" and "^00.;1"

Further, on VMS, "." and ".." aren't special either. These files show as ".;1" and "^..;1".

These exceptions may lead to confusion or problems. At the moment they aren't supported in the ods5 file system. Again, such filenames are rare.

Please note, the ODS-5 docs say, that for ISL-1 the ISO Latin-1 character set is used. However, some users may have had the DEC MCS in mind and enabled for their terminal (DECterm or VTnnn), when creating the file. These character sets only differ in a few entries. For these, the user may expect different filenames. It is unlikely that this will happen very often but obviously, this is not a problem of the ods5 file system. Offering an additional not even supported character set, DEC MCS, is not really an option.

It's not obvious and can not be checked from the file header/directory entry, which character set (ISO Latin-1/DEC MCS) was used when the filename was entered.

In a strict VMS environment, it is very likely DEC MCS, in a mixed environment with terminal emulators running on other operating systems and or NFS/ftp, archives etc. it very likely is ISO Latin-1.

A new option displaying 8-bit ISO Latin-1 characters with escape sequences seems to be a compromise. This may be implemented with escape sequences for non-printable ASCII characters. An option like escape=ctrl escape=8bit, where the first value seems useful for utf8 and vtf7, and the second only useful for vtf7 with no ISO Latin-1 character set loaded.

The files in a directory are sorted. To list them in the VMS sort order the option -U for the ls command is recommended. As pointed out in another web page, the ODS-5 sort is caseblind in uppercase. Therefore 'x' comes before 'Y'. This is also true for ISL-1 filenames: 'Ä' is immediately followed by 'ä' and then by 'Å'. UCS-2 filenames are sorted "case blind" as well. Consider U+03A9 (the greek capital letter omega, 'Ω'), U+03AA (the greek capital letter iota with dialytika, 'Ϊ') and U+03C9 (the greek small letter omega, 'ω'): as these are escaped on the command line one sees them ordered in the directory as ^U03A9, ^U03C9 and ^U03AA (or in the ods5 file system with utf8 support as 'Ω', 'ω' and 'Ϊ'). (Other Unicode pages have more intuitive code points, for example U+0100 and U+0101 are the latin capital and small letter a with macron, 'Ā' and 'ā'. Seeing the escape sequences ^U0100 followed by ^U0101 is more like the sort order which a user might expect.

Additionally if there is a mix of ISL-1 and UCS-2 filenames in a directory, the sort order may not be what one expects. Consider the (ISL-1) character 0xff ('ÿ') and the UCS-2 character U+0100 ('Ā'), for unknown reasons (at least to me) the order in the directory is ^U0100 followed by ^FF. (I would have expected to have them "case blind" sorted by Unicode code points, which would be U+00FF followed by U+0100.)

As noted before, VMS on the command line prints ^FF although this is a printable character.

VMS/ODS-5 symbolic links are supported as well. The links are ISO Latin-1 strings. Which on one side means, you can't have UCS-2 files as target. On the other hand, it requires the ods5 file system to convert that string to UTF-8 as well. Only internally and shown to the user, not on the disk. With UTF-8 requiring more space than ISO Latin-1, the internal file size of symbolic links is adjusted. So when the disk is mounted with utf8 the user may see with "ls -l" a bigger file size of a symbolic link than when the same disk is mounted with vtf7. 
# ls -l /vms/isl1.DIR\;1/a*
lrwx------ 1 8247 392 7 Dec 22 15:26 /vms/isl1.DIR;1/ahem.;1 -> xä.txt
versus 
# ls -l /vms/isl1.DIR\;1/a*
lrwx------ 1 8247 392 6 Dec 22 15:26 /vms/isl1.DIR;1/ahem.;1 -> xä.txt
Please note, on VMS versions prior to V8.4, symbolic links don't work for target filenames with 8-bit ISO Latin-1 characters (and link filenames are created in uppercase characters although not specified as such): 
$ copy tt: tä.txt
target, name with ISO Latin-1 character: ä
$
$ cre/syml="tä.txt" symlink
$ dir/size

Directory USR_ODS5:[USER]

SYMLINK.;1 -> tä.txt
                           1
tä.txt;1                   1
			   
Total of 2 files, 2 blocks.
$
$ ty symlink.
%TYPE-W-OPENIN, error opening USR_ODS5:[USER]SYMLINK.;1 as input
-RMS-E-DNF, directory not found
-SYSTEM-W-NOSUCHFILE, no such file
$