Use CReMA JUICE SKD version to ensure the reproductibility of the results
JUICE metakernel SKD version
Since Nov. 2020 the JUICE meta-kernels include a SKD_VERSION = ‘vXXX_YYYMMDD_YYY’
key (eg. v282_20210505_001
) which correspond to the published tag on the JUICE SPICE repo.
This value is added to the SPICE Pool with the SPICE furnsh
function when a meta-kernel is loaded:
>>> import spiceypy
>>> spiceypy.kclear()
>>> spiceypy.furnsh('juice_crema_3_0.tm')
>>> spiceypy.ktotal('ALL')
56
>>> spiceypy.gcpool('SKD_VERSION', 0, 100)
['v282_20210505_001']
This value can be use to ensure the reproductibility of the SPICE computation on a given set of kernels.
Missing additional data
To ensure the re-definition of the PATH_VALUES
at runtime, the moon_coverage.kernels.MetaKernel
load the data and only provide the list of kernels to load to the spiceypy.furnsh
function (as an iterable). Any additionnal data are lost in the process.
Possible solutions
- Load the additional values after the furnsh call (this could have side effects if the user implement a custom important and forget to add the additional data).
- The metakernel shoud be edited on-the-fly in a in-memory/temp file and
furnsh
into the pool as a regular text-file.
Locate the latest metakernel version
We need a method to get the latest version number for a specific mission:
>>> get_latest('JUICE')
'v282'
Currently there is different way to get the latest version id:
- as a
json
output from the Bitbucket API. - as a raw txt file from the root of the SPICE JUICE repo.
- as a misc text file from the SPICE FTP server.
- with a git request with
ls-remote
:
git ls-remote --tags https://repos.cosmos.esa.int/socci/scm/spice_kernels/juice.git
In all cases, if no internet access is available the function should throw an error:
>>> get_latest('JUICE')
IOError: Impossible to locate the latest version on `https://repos.cosmos.esa.int/socci/projects/SPICE_KERNELS/repos/juice/`.
Support for custom endpoint should be available:
>>> get_latest('https://repos.cosmos.esa.int/socci/projects/SPICE_KERNELS/repos/bepicolombo/')
Possible issues
-
json
query can be very inexpensive and quick to retrive. It does not require disk access but it dependent of the API response schema. A small parsing step is required to convert the tag into a version number. - by default this will require to download the file on disk (maybe not) but it contains directly the version key without any parsing step and it is not API reponse schema dependant.
- same as
2
but it is hosted on aftp
server and not ahttps
one (security drawback and it is not possible to inspect the file before downloading it). It might also not be in-sync with the latest version on the repo (considered to be the main entry point by the SPICE Service). - it will require that the user have a
git
instence install locally (which is not always the case). It also require to parse the tag as in1
.
CReMA versionning
The moon_coverage.esa.CReMACollection
should be able to re-write the metakernel url based on its version:
>>> JUICE_CReMA['4.2b22.1', 'v282']
<JUICE_CReMA> juice_crema_4_2b22_1_v282.tm
>>> JUICE_CReMA['4.2b22.1', 'v282'].fname
'moon_coverage/esa/data/juice_crema_4_2b22_1_v282.tm'
>>> JUICE_CReMA['4.2b22.1', 'v282'].version
'v282'
>>> JUICE_CReMA['4.2b22.1', 'v282'].url
'ftp://spiftp.esac.esa.int/data/SPICE/JUICE/kernels/mk/juice_crema_4_2b22_1_v282.tm'
>>> JUICE_CReMA['4.2b22.1', 'v281'].url
'ftp://spiftp.esac.esa.int/data/SPICE/JUICE/kernels/mk/former_versions/juice_crema_4_2b22_1_v281.tm'
It should check if the file is present locally before checking that it exists online on the spice FTP server (and download it if it's missing).
If the version provided is invalid (not present locally or nor on the FTP server), an error should be thrown:
>>> JUICE_CReMA['4.2b22.1', 'v999']
KeyError: The version requested `v999` is larger than the latest available version (`v282`).
If no version is provided, the tool should always search for the latest version available locally present:
>>> JUICE_CReMA['4.2b22.1'].fname
'moon_coverage/esa/data/juice_crema_4_2b22_1_v282.tm'
>>> JUICE_CReMA['4.2b22.1', None].fname
'moon_coverage/esa/data/juice_crema_4_2b22_1_v282.tm'
The key latest
should also be considered as a valid key to always check and get the latest version:
>>> JUICE_CReMA['4.2b22.1', 'latest'].url
'ftp://spiftp.esac.esa.int/data/SPICE/JUICE/kernels/mk/juice_crema_4_2b22_1_v282.tm'
The version key should be not sensitive the v
prefix:
>>> JUICE_CReMA['4.2b22.1', 282].url
'ftp://spiftp.esac.esa.int/data/SPICE/JUICE/kernels/mk/juice_crema_4_2b22_1_v282.tm'
or the dotted version:
>>> JUICE_CReMA['4.2b22.1', '2.8.2'].url
'ftp://spiftp.esac.esa.int/data/SPICE/JUICE/kernels/mk/juice_crema_4_2b22_1_v282.tm'
>>> JUICE_CReMA['4.2b22.1', 'v2.8.2'].url
'ftp://spiftp.esac.esa.int/data/SPICE/JUICE/kernels/mk/juice_crema_4_2b22_1_v282.tm'
Possible issues
- Currenlty the latest version of the kernel is not archived in the
kernels/mk/former_versions/
folder but one level above (this issus was reported to the SPICE Service on April 13th 2021). - The JUICE SPICE webpage currently point to the non-tagged versions of the metakernels and no explicit information is provided regarding the actual SKD version used (this issue was also reported to the SPICE Service on April 13th 2021). This endpoint should be used only as an initial description of the kernels.
- It may be relevant to parse the description of the meta-kernel directly from their header and not from the webpage.
- If no version is provided, the latest local version will be use. It might not be the latest version version available online. The user is expected to provide an explicit
latest
key to get the latest version from the SPICE repo. This would ensure that tool remain resilient if no connexion to the SPICE JUICE repo is not available.
TourConfig SKD version
The moon_coverage.trajetcory.TourConfig
should have an optional version
argument that will point to the correct metakernel version
and expose the SKD_VERSION
when it is present in the pool.
>>> tour = TourConfig(mk='4.2b22.1', version='v282')
<TourConfig> Spacecraft: JUICE | Target: GANYMEDE | Metakernel: juice_crema_4_2b22_1_v282.tm | SKD: v282_20210505_001
>>> tour.version
'v282'
>>> tour.skd
'v282_20210505_001'
By default, the version
argument should be an optional attribute (default: None
) but the user should be able to force the tool to always use the latest version:
>>> TourConfig(mk='4.2b22.1', version='latest')
<TourConfig> Spacecraft: JUICE | Target: GANYMEDE | Metakernel: juice_crema_4_2b22_1_v282.tm | SKD: v282_20210505_001
Possible issue
- Use of the
version=latest
flag can result in a delay in the kernel loading process (because the latest online version will be query each time).
Additional notes
- Currently the dependency are weaking defined in the
setup.py
. To ensure a better repoductibility of the notebooks, we need to freeze their version more precisely. - The flag
--nbval-lax
in thedocker/jupyter-lab.tests.dockerfile
could be removed to ensure an exact reproduction of the notebook during the docker CI testing.
To do
-
Fix MetaKernel
additional data loader. -
Add latest version checker. -
Add version item in CReMACollection
. -
Add SKD version support in TourConfig
-
Improve kernel pool versioning. -
Update the docs accordingly -
Update examples notebook. -
Update Dockerfile
-
Pick more precisely the dependencies in setup.py
(optional)