Data Handling#

cherab-lhd handles LHD-related data like EMC3-EIRENE grid configuration, machine geometry, measurement geometry, etc. Here is a brief description of the dataset containing the above-mentioned data.

Dataset Registries#

cherab-lhd provides a list of hash values of the files instead of providing the data files themselves. fetch module offers functionalities to manage the download of the data files by checking if the files are not present in the cache directory.

[1]:

from cherab.lhd.tools.fetch import show_registries

show_registries()

                                           Registries                                            
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ File Name                  ┃ Hash Value                                                       ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ emc3/grid-360.nc           │ 948c65fbefc78118272e223720930d10846b1b177f513b4cadfc33ec9a177f3d │
├────────────────────────────┼──────────────────────────────────────────────────────────────────┤
│ grid-demo.nc               │ md5:34b5f81fc47d116e7318219b6f7a749f                             │
├────────────────────────────┼──────────────────────────────────────────────────────────────────┤
│ machine/calcam_cad.ccm     │ a93db1a11d16357ccbfe797bf71d7d78aaf008a9e95e6c0aa15a7bb1207cb528 │
├────────────────────────────┼──────────────────────────────────────────────────────────────────┤
│ machine/divertor.rsm       │ 6ae9b4b50139818b7e600a247bb4fc3d1260bac28a60234a60fbd6c1aebdc1ca │
├────────────────────────────┼──────────────────────────────────────────────────────────────────┤
│ machine/plasma.rsm         │ ae950c8d787fad3ebccbfef12f2282cd927d15a6cc9554db28c9d39a4f7eda38 │
├────────────────────────────┼──────────────────────────────────────────────────────────────────┤
│ machine/plasma_cut.rsm     │ 10fb5fc23ccc490b265a12e8e77b962a3cd4c8b899608bb30de6ce0753d45ea0 │
├────────────────────────────┼──────────────────────────────────────────────────────────────────┤
│ machine/plasma_quarter.rsm │ d517d96e690fd458e90379d0e9e6e06e4da8793986f28d636817d59b1eb9171d │
├────────────────────────────┼──────────────────────────────────────────────────────────────────┤
│ machine/plates.rsm         │ d479746123d071c5070b5942ca3a5d53fbe07dda73d7651e64e951c04fbf33c7 │
├────────────────────────────┼──────────────────────────────────────────────────────────────────┤
│ machine/port_65l.rsm       │ cd0debab846d86d097d3b66007496e13b7b8fc76896c102b758a4f47478e3adb │
├────────────────────────────┼──────────────────────────────────────────────────────────────────┤
│ machine/port_65u.rsm       │ 46ad570bd222b7ceae8f2098c65ad2a45d53ef95845b4ac75be221acf60b057c │
├────────────────────────────┼──────────────────────────────────────────────────────────────────┤
│ machine/vessel.rsm         │ ad90ab5b596c317a329cb7820d46da2f54fcb844476bae1abe0203dc383a89bf │
├────────────────────────────┼──────────────────────────────────────────────────────────────────┤
│ machine/wall_outline.nc    │ 77883c34bb89bcf909bb15d612e750b58194726a87cf94fe310a2e80bba625e3 │
├────────────────────────────┼──────────────────────────────────────────────────────────────────┤
│ material/sus316L.json      │ 0f473be0d3f9efc88d11d3cbacbc5a8596ff2b39dcb2dbd561d0fa116de5f301 │
├────────────────────────────┼──────────────────────────────────────────────────────────────────┤
│ observer/IRVB.json         │ a46010c1d67bbe71c0bf71c203447cb984c5622e63e10d20f5ee8b8d2cf93248 │
├────────────────────────────┼──────────────────────────────────────────────────────────────────┤
│ observer/RB.json           │ bffd237b98bc6a158fb4f4875d1d79d6571fe4770561f8e18fd1da72a63bcae2 │
├────────────────────────────┼──────────────────────────────────────────────────────────────────┤
│ tetra/zone0+zone11.rsm     │ 719f69ae4ff3d94121a4de4efd45ac9116988ded30786cdf70571df199c9d416 │
├────────────────────────────┼──────────────────────────────────────────────────────────────────┤
│ tetra/zone0.rsm            │ 2859a0618badd7d764691750bbc617479a00ee4f0a31dcf9d616d81174f3402b │
├────────────────────────────┼──────────────────────────────────────────────────────────────────┤
│ tetra/zone1.rsm            │ a097e6f5dcf0319e3109fc0d7c010442297e6078778cbc795bc37729ff6fc2be │
├────────────────────────────┼──────────────────────────────────────────────────────────────────┤
│ tetra/zone11.rsm           │ dc45ebe22b858db13f1cd73e43422629dde6adc2f8ca2a4c14cb8e4a2b83dd14 │
├────────────────────────────┼──────────────────────────────────────────────────────────────────┤
│ tetra/zone12.rsm           │ a16ef6d09ad1f7551280d405d91aa63021b5eac6e0e397e563ee3130ed52c98e │
├────────────────────────────┼──────────────────────────────────────────────────────────────────┤
│ tetra/zone13.rsm           │ 8ded151d42952c3b83b9293d9c3e4be28ebe847608d7e8707d8daf518063949f │
├────────────────────────────┼──────────────────────────────────────────────────────────────────┤
│ tetra/zone14.rsm           │ 3ab9bfd2e8c2b6631c269a02a1597deebde1fce973ab61f3db9065054ced5538 │
├────────────────────────────┼──────────────────────────────────────────────────────────────────┤
│ tetra/zone15.rsm           │ 25381a62eda6c217f280e0bf8d93fe077bb1c517a1e3dfa1fc60043db5be5fa2 │
├────────────────────────────┼──────────────────────────────────────────────────────────────────┤
│ tetra/zone2.rsm            │ 33f22ac916633281e7c246f1092b24e41e2b9ca9d4497793c2c87a349bad526a │
├────────────────────────────┼──────────────────────────────────────────────────────────────────┤
│ tetra/zone3.rsm            │ 0e9178decb85664309a5de2b6d0d1ea3df0866ecface573407c276f05547bad3 │
├────────────────────────────┼──────────────────────────────────────────────────────────────────┤
│ tetra/zone4.rsm            │ 42c5c1469d9fd4acb14e5dbb581dda570804ce84f232c70178feb42cab0681cc │
└────────────────────────────┴──────────────────────────────────────────────────────────────────┘

Retrieving the Dataset#

SFTP download#

Most datasets are downloaded via the SFTP protocol from the remote server if they are not already available in the local cache directory or if their hash differs from the one in the registry.

Before downloading the dataset, users need to have access to the remote server. All datasets are not yet publicly available. Those who want to access the dataset must contact the package maintainers.

After getting permission to deal with the dataset, users need to set the following environment variables:

SSH_RAYTRACE_HOSTNAME: The hostname of the remote server (e.g. sftp://hostname/directory/).
SSH_RAYTRACE_USERNAME: The username to log in to the remote server.
SSH_RAYTRACE_PASSWORD: The password to log in to the remote server.

Then, use the fetch_file function:

>>> from cherab.lhd.tools.fetch import fetch_file
>>> path = fetch_file("emc3/grid-360.nc")

Or, manually specify them as arguments to the fetch_file function.

>>> path = fetch_file("emc3/grid-360.nc", host="sftp://hostname/directory/", username="username", password="password")

Downloaded files are stored in the user’s cache directory:

Linux: ~/.cache/cherab/lhd/
macOS: ~/Library/Caches/cherab/lhd/

If the dataset is no longer needed, please delete the downloaded files manually.

Other download methods#

Some datasets are publicly available via DOI links. URL links to the corresponding data can be accessed by using the get_urls method:

[2]:

from cherab.lhd.tools.fetch import get_urls

get_urls()

[2]:

{'grid-demo.nc': 'doi:10.5281/zenodo.15412887/grid-demo.nc'}

The path to the publicly available dataset can be also retrieved like this:

>>> path = fetch_file("grid-demo.nc")

Locate the Dataset file manually#

Users can put dataset files in the cache directory manually. Please confirm the cache directory depending on the operating system as specified above and place the files under the cherab/lhd directory with corresponding subdirectories (e.g. emc3, machine, observer, etc.)

Note that even if the files are placed locally, they will be downloaded from the remote server if the hash value is different from the one in the registry.