IBM zOS and OS 400 Support
FluentFTP has extensive support for all variants of IBM z/OS and OS/400 Servers.
All credits for the development and testing of our excellent IBM z/OS API go to Michael Stiemke.
z/OS, also historically known as MVS, is only one of IBM's four mainframe operating systems (z/OS, z/VM, z/VSE, z/TPF).
One of the big stumbling blocks for accessing IBM OS/400 FTP servers is that, in a similar fashion to z/OS servers, there are a number of different listing formats that can be configured.
Beyond the somewhat strange naming schemes and the capability to directly access databases instead of just files, there is significantly less complexity to handle.
Most users in the past have probably used the library/filename format, which is more or less the default on these systems. Currently FluentFTP will not set Fullname correctly in this mode. Users will use FtpListOptions.Nopath, having navigated directly to the library where they plan to list files and up/download files.
If you want to have more control, consider reading this and perhaps using the SITE command to set LISTFMT to 1. In some cases this can improve your GetListing experience.
Returns the true file size in bytes for any z/OS file system object.
Returns a value of the enum FtpZOSListRealm to describe the z/OS realm of the current working directory.
Return values can be:
-
FtpZOSListRealm.Invalid- Current server type and OS are not z/OS. -
FtpZOSListRealm.Unixwhen the current working directory is in the unix (HFS / USS) realm. -
FtpZOSListRealm.Datasetwhen the current working directory is in the general z/OS dataset realm. Note: This value is returned both for directories that represent an existing dataset ( which is then by nature sequential (DSROG PS)) or a directory that is not yet fully qualified . -
FtpZOSListRealm.MemberThe current working directory fully qualifies a partitioned dataset. You could now list the members. -
FtpZOSListRealm.MemberUThe current working directory fully qualifies a partitioned dataset and it is a RECFM=U dataset. You could now list the members.
Check if the current working directory is the root directory of the file system. (Lexical checks are done to determine it)
Example:
client.SetWorkingDirectory("'GEEK.PRODUCT.LLIB'");
bool isr = client.IsRoot();There are 2 realms that FluentFTP supports, and here is some example code for both realms:
Native z/OS realm:
string zosPath = "'SYS1.'"; // you can also use 'SYS1', both work
client.SetWorkingDirectory(zosPath);
FtpListItem[] list = client.GetListing("", FtpListOption.NoPath);
string fullName = zosPath.Trim('\'').Trim('.') + '.' + list[0].FullName.Remove(0, 1);USS (linux-like) realm:
string zosPath = "/projects";
FtpListItem[] list = client.GetListing(zosPath);
string fullName = list[0].FullName;In z/OS, using the Communication Server for z/OS, the FTP server can LIST the following types of entries:
- Classic z/OS datasets (many still refer to these as MVS datasets).
- Datasets in other mounted filesystems. These are often referred to as HFS (Hierarchical File System) files, some people refer to these as files in USS (Unix System Services).
- Members of classic z/OS partitioned non-RECFM=U datasets
- Members of classic z/OS partitioned RECFM=U datasets (load libraries)
When you call GetListing, there are 4 types of listing that are implemented in order to handle variants of the z/OS filesytems:
- List of USS files, is unix-like and handled by the standard FluentFTP unix parser
- List of normal z/OS datasets (non VSAM)
- List of members of partitioned datasets (non load library)
- List of members of partitioned datasets (load library)
This IBM document documents the 4 different output formats of the LIST (or DIR) command.
The following test program demonstrates the differences between the various z/OS filesytems:
FtpClient client = new FtpClient();
client.Host = "ftps://xxxxxxx.xxxxx.com";
client.Credentials.UserName = "myuser";
client.Credentials.Password = "secret";
client.ValidateAnyCertificate = false;
client.ValidateCertificate += new FtpSslValidation(OnValidateCertificate);
client.AutoConnect();
// Member Listing non loadlib
string zosPath = "'MYUSER.PRODUCT.CLIB'";
client.SetWorkingDirectory(zosPath);
FtpListItem[] list1 = client.GetListing("", FtpListOption.NoPath);
// Member listing loadlib
zosPath = "'SYS1.V1R3M0.SEAGALT'";
client.SetWorkingDirectory(zosPath);
FtpListItem[] list2 = client.GetListing("", FtpListOption.NoPath);
// Dataset listing
zosPath = "'MYUSER.'";
client.SetWorkingDirectory(zosPath);
FtpListItem[] list3 = client.GetListing("", FtpListOption.NoPath);
// USS file listing
zosPath = "/projects";
FtpListItem[] list4 = client.GetListing(zosPath);
client.Disconnect();Alternative #1
You can use GetListing() with an empty path and with FtpListOption.NoPath, which causes FluentFTP to use the LIST command without any further parameters - the zOS FTP Server accepts this in the z/OS realm and in the HFS / USS realm.
This in turn means that you must SetWorkingDirectory() to go to the desired working directory before doing the GetListing("", FtpListOption.NoPath.
string zosPath = "'MYUSER.PRODUCT.CLIB'";
client.SetWorkingDirectory(zosPath);
FtpListItem[] list1 = client.GetListing("", FtpListOption.NoPath);Alternative #2
You can use GetListing() with a z/OS realm path and without FtpListOption.NoPath. If the dataset exists, you will get a list with one entry. Otherwise your list will be empty.
To list all datasets below a partly qualified z/OS realm path, you can append *. To list all members of a partitioned dataset, append (*).
Samples
// HFS with path
FtpListItem[] Listu1 = client.GetListing("/projects");
// HFS with NoPath option
client.SetWorkingDirectory("/projects");
FtpListItem[] Listu2 = client.GetListing("", FtpListOption.NoPath);
// z/OS with path
FtpListItem[] Lista1 = client.GetListing("PRODUCTS.*");
FtpListItem[] Lista2 = client.GetListing("PRODUCTS.CLIB(*)");
FtpListItem[] Lista3 = client.GetListing("PRODUCTS.CLIB");
FtpListItem[] Lista4 = client.GetListing("'GEEK.PRODUCTS.*'");
FtpListItem[] Lista5 = client.GetListing("'GEEK.PRODUCTS.CLIB(*)'");
FtpListItem[] Lista6 = client.GetListing("'GEEK.PRODUCTS.CLIB'");
// z/OS with NoPath option
client.SetWorkingDirectory("'GEEK'");
client.SetWorkingDirectory("PRODUCTS");
FtpListItem[] Listb1 = client.GetListing("", FtpListOption.NoPath);
client.SetWorkingDirectory("PRODUCTS.CLIB");
FtpListItem[] Listb2 = client.GetListing("", FtpListOption.NoPath);.FullName is now set correctly in all these scenarios.
Additional notes
The IBM CS FTP server default behaviour/setup as configured might make GetListing(...) fail. Make sure the following settings are in effect:
DIRECTORYMODE FALSE
QUOTESOVERRIDE TRUE
These settings are the default. You could change them in the FTP server configuration and if this is not possible for you, you can set these in the FTP session dynamically.
You want:
client.Execute("SITE DATASETMODE")
client.Execute("SITE QUOTESOVERRIDE")You can check if these settings are in effect by using a STAT command. Search for these lines in the answer:
stat
>>> STAT
...
211-Data set mode. (Do not treat each qualifier as a directory.)
...
211-Single quotes will override the current working directory.
...
FluentFTP will attempt to configure these settings on connection automatically if it detects the IBM CS FTP z/OS server.
It will also try to set SITE LISTLEVEL=0 (if supported) to get away from any SITE LISTLEVEL=1 that might be active, and then if supported, try to set SITE LISTLEVEL=2, again if supported. Once again, you can check the current setting yourself by issuing a STAT command.
In case your IBM CS FTP server reports that is supports features like SIZE, MDTM and UTF8, this is fine but be aware that these features are only relevant to FTP actions against unix (HFS) files.
Use this to set up host file paths from GetListing()results:
Note that this example uses Alternative #1
FtpClient client = new FtpClient();
client.Host = "ftps://xxxxxxx.xxxxx.com";
client.Credentials.UserName = "myuser";
client.Credentials.Password = "secret";
client.ValidateAnyCertificate = false;
client.ValidateCertificate += new FtpSslValidation(OnValidateCertificate);
client.AutoConnect();
string FTP_Hostfile;
string cwd = "'GEEK.PRODUCTS.LOADLIB'"; // Known to exist, PDS
client.SetWorkingDirectory(cwd); // Position to there
FtpListItem[] entries = client.GetListing("", FtpListOption.NoPath);
FtpListItem entry = entries[0]; // as an example, take the first entry.
if (cwd[0] == '\'') // z/OS realm
{
// Check entry type. If it is not "Dataset", it must be Member or MemberU
if (client.GetZOSListRealm() != FtpZOSListRealm.Dataset) //
// remove quotes, add "(membername from GetListing())" and requote
FTP_Hostfile = "'" + cwd.Trim('\'') + "(" + entry.Name + ")'";
else
// remove quotes, add "(name from GetListing())" and requote
FTP_Hostfile = "'" + cwd.Trim('\'') + entry.Name + "'";
}
else // HFS / USS realm
FTP_Hostfile = cwd + entry.Name;
// and now do something with FTP_Hostfile, perhaps download or upload
client.Disconnect();To find out where you are, use GetWorkingDirectory().
string pwd = client.GetWorkingDirectory();
// pwd examples:
// 'GEEK.PRODUCTS.LOADLIB' This dataset exists.
// 'GEEK.PRODUCTS.' This is not an existing dataset. Note the trailing period - it tells us this fact
// /projects This is a HFS / USS path. Note there are no single quotes, but a slash in front alwaysTo check if you are already "up at the top", use IsRoot(). It knows about z/OS dataset names and would consider a name such as 'SYS1.' to be a root. Any HLQ (high level qualifier on its own is considered to be root.
You can use SetWorkingDirectory("..") from such a root, but there no way to get a listing from there.
Get the result of a PWD and check the first character. If it is a single quote, it is the classic z/OS dataset realm, otherwise expect unix-like paths and filenames.
You can use GetZOSListRealm() to do exactly that. You will be given FtpZOSListRealm.Unix, FtpZOSListRealm.Dataset, FtpZOSListRealm.Member or FtpZOSListRealm.MemberU
Note that this tells you how the z/OS GetListing "directory" parser would parse the listing if you were to call GetListing("", FtpListOption.NoPath) at that location. So a result of FtpZOSListRealm.Member does not mean that the current working directory is a member (of a PDS). It rather means, you will get a list of members and that the current working directory is a partitioned dataset.
Since the z/OS CS FTP server does not support the SIZE command to get the file size, here is a quick way to get it (only works if the host file is known to exist):
long size = client.GetFileSize("'GEEK.PRODUCTS.LOADLIB(FLXWTO)'");It handles all the slash, single-quote, parentheses mangling that is needed for z/OS realms. This file size is subject to the same caveats as noted in the file size table further down.
Note that there is a reason for a missing indication of an exact actual file size information in native z/OS file systems (realms): It is a totally different architecture, even down to the physical disk layout. As stated elsewhere, the actual amount of bytes consumed when downloading such a z/OS dataset can not be predicted accurately by any other way than actually downloading it or reading it host-side. To repeat: This is a totally different approach to file allocation, reserve extents, compression, block and record organisation and in the end, you get a totally different concept of actual data use as a percentage of reserved and thus space unavailable to others. You allocate a dataset with certain constraints, and then use this space to the extent you decide on, under the limits you constrained yourself to upon allocation. You have been given this allocation and may use as much of it as you please. You are billed the allocation, not the actual use of it. So who cares how full it is?
The code will supply a size for each entry according to the following table:
HFS (or USS) realm: Size is correct.
z/OS (aka MVS) datasets: PS (i.e. non VSAM, non PDS, just sequential): Size is calculated by multiplying the used tracks by 3390-bytes-per-track (56664 bytes). Depending on the RECFM (consider for example V, VB, VBA), and/or compression, this might overestimate the size by a large amount. Best results on RECFM F, FB, FBA.
PO (i.e. Partitioned Dataset): Size is calculated by multiplying the used tracks by 3390-bytes-per-track (56664 bytes). Depending on the RECFM (consider for example V, VB, VBA), and/or compression, this might overestimate the size by a large amount. Best results on RECFM F, FB, FBA.
Members of a partitioned dataset:
non-RECFM=U (for example: RECFM=FB, very common): Size is returned as "number of records" multplied by LRECL. Depending on the RECFM (consider for example V, VB, VBA), and/or compression, this might overestimate the size by a large amount. Best results on RECFM F, FB, FBA.
LRECL is determined by issuing an internal XDSS command (see below).
Note that these file sizes depend on ISPF stats stored in the directory of the PDS for each member. These may be missing if the member was created by a non ISPF program. The size will then be reported as being zero. To create stats for such members, use a host-side program that invokes ISPF lib mngmt. service LMSTATS.
RECFM=U: Size is returned from the hex length field of the load library listing format and is correct
Please refer to PR #765
Not tricky at all: Since the introduction of MVSPUT and MVSGET FTP subcommands (from z/OS 2.1 onwards), there is a virtually undocumented command XDSS, which is needed by the functionality of these new commands.
Here is an example:
XDSS 'GEEK.TESTPROG.ASM'
200-LASTREF=2021/10/18 DSEMPTY=FALSE
200 SITE PDSTYPE=PDSE RECFM=FB BLKSIZE=16000 DIRECTORY=1 LRECL=80 PRIMARY=3 SECONDARY=110 TRACKS EATTR=SYSTEM
Note that you will very likely not be able to enter the XDSS FTP subcommand at your favorite ftp client. If you want test it, you need to use it from your FluentFTP session programatically, like in this snippet:
long LRECL;
if (entry.Type == FtpFileSystemObjectType.Directory)
{
string pwd = GetPWD();
if (pwd[0] == '\'')
{
LRECL = 0;
FtpReply response = client.Execute("XDSS '" + pwd.Trim('\'') + entry.Name + "'");
if (response.Success)
{
// SITE PDSTYPE=PDSE RECFM=FB BLKSIZE=16000 DIRECTORY=1 LRECL=80 PRIMARY=3 SECONDARY=110 TRACKS EATTR=SYSTEM
string[] words = response.Message.Split(' ');
string[] val = words[5].Split('=');
LRECL = Int64.Parse(val[1]);
}
InPDS = true;
}
}When SITE FILETYPE=JES is active (as opposed to SITE FILETYPE=SEQ), a
-
STOR filenamewill upload a jcl file from your client system to the host and submit this as a job. -
LISTwill list the job output queue, from which you can select a job output to download to your client system -
RETR jobname.nwill then download the jobs output file number n.
Here is an example:
FTP_Sess.Config.ListingDataType = FtpDataType.ASCII;
FTP_Sess.Config.UploadDataType = FtpDataType.ASCII;
FTP_Sess.Execute("SITE FILETYPE=JES");
FTP_Sess.UploadFile("D:\\temp\\JES\\LISTCAT", "LISTCAT", FtpRemoteExists.NoCheck, false, FtpVerify.None);
assuming you have a file on your system with some jcl. This upload will transfer the jcl from the file to the JES2 internal reader and submit the job.
The LastReply object after the upload will look like this:
You will see, in the InfoMessages, there is 250-It is known to JES as JOBnnnnn.
So then you do:
FTP_Sess.Config.ListingDataType = FtpDataType.ASCII;
FTP_Sess.Config.DownloadDataType = FtpDataType.ASCII;
FtpStatus Jest = FTP_Sess.DownloadFile("D:\\temp\\JES\\LISTCAT.OUT.1", "JOB00061.1", FtpLocalExists.Overwrite, FtpVerify.None);
Here is the complete log of the process:
Command: SITE FILETYPE=JES
Status: Waiting for response to: SITE FILETYPE=JES
Response: 200 SITE command was accepted
> UploadFile("D:\temp\JES\LISTCAT", "LISTCAT", NoCheck, False, None)
> OpenWrite("LISTCAT", ASCII)
> OpenPassiveDataStream(AutoPassive, "STOR LISTCAT", 0)
Command: EPSV
Status: Waiting for response to: EPSV
Response: 229 Entering Extended Passive Mode (|||1913|)
Status: Connecting to ***:1913
Command: STOR LISTCAT
Status: Waiting for response to: STOR LISTCAT
Response: 125 Sending Job to JES internal reader FIXrecfm 80
Status: Disposing FtpSocketStream...
Status: Waiting for a response
Response: 250-It is known to JES as JOB00060
Response: 250 Transfer completed successfully.
> DownloadFile("D:\temp\JES\LISTCAT.OUT.1", "JOB00061.1", Overwrite, None)
> OpenRead("JOB00061.1", ASCII, 0, 0)
Command: TYPE A
Status: Waiting for response to: TYPE A
Response: 200 Representation type is Ascii NonPrint
> OpenPassiveDataStream(AutoPassive, "RETR JOB00061.1", 0)
Command: EPSV
Status: Waiting for response to: EPSV
Response: 229 Entering Extended Passive Mode (|||1915|)
Status: Connecting to ***:1915
Command: RETR JOB00060.1
Status: Waiting for response to: RETR JOB00061.1
Response: 125 Sending data set BFSYS.BFSYSA.JOB00061.D0000002.JESMSGLG
Status: Disposing FtpSocketStream...
Status: Waiting for a response
Response: 250 Transfer completed successfully.
You can also do:
FTP_Sess.Config.ListingDataType = FtpDataType.ASCII;
FtpListItem[] jesList = FTP_Sess.GetListing("LIST", FtpListOption.ForceList | FtpListOption.NoPath | FtpListOption.NoImage);
var listLine = jesList[0].Input;
which would give you in listLine: "BFSYSA JOB00069 BFSYS OUTPUT A RC=0004 4 spool files ". Parsing that will tell you there are 4 files, JOB00061.1 through JOB00061.4 that you could retrieve.