Issues in Data Preparation and Modelling

[abhisheksinghIN]

1. After implementing sentinel2_prepare.py;

2. But during implementing meraner_processor.py, the following issues are;

root@3dfcdc1d2ec0:/workspace/decloud/decloud/production# python meraner_processor.py --in_s2 workspace/decloud/data/S2_PREPARE/T32TPS/ --in_dem workspace/decloud/data/DEM_PREPARE/T32TPS.tif --output workspace/decloud/data/output/ --savedmodel workspace/decloud/data/model/merunet_occitanie_pretrained.zip
2024-06-24 08:49:39 (INFO) [pyOTB] Successfully loaded 126 OTB applications
2024-06-24 08:49:39 INFO Init. S2_TILED product
2024-06-24 08:49:39 INFO Product is not of type s2 (Exception: Not a S2_TILED product (expected 1 files ending with _FRE_10m.tif))
2024-06-24 08:49:39 INFO Init. S2_THEIA product
2024-06-24 08:49:39 INFO Input type is a directory
2024-06-24 08:49:39 INFO Product is not of type s2 (Exception: workspace/decloud/data/S2_PREPARE/T32TPS/ not a S2_THEIA product : _FRE_B2.tif is mi
2024-06-24 08:49:39 INFO Init. S2_ESA product
2024-06-24 08:49:39 INFO Input type is a directory
2024-06-24 08:49:39 INFO Product is not of type s2 (Exception: workspace/decloud/data/S2_PREPARE/T32TPS/ not a S2_ESA product : _B02_10m.jp2 is mis
Traceback (most recent call last):
File "/workspace/decloud/decloud/production/meraner_processor.py", line 223, in
sys.exit(main())
File "/workspace/decloud/decloud/production/meraner_processor.py", line 217, in main
meraner_processor(params.il_s1, params.in_s2, params.savedmodel, params.in_dem,
File "/workspace/decloud/decloud/production/meraner_processor.py", line 73, in meraner_processor
s2t = s2t_product.get_raster_10m()
AttributeError: 'NoneType' object has no attribute 'get_raster_10m'

[abhisheksinghIN]

After S1_prepare.py; data directory: workspace/decloud/data/S2_PREPARE/T32TPS/SENTINEL2A_20230821-101805-444_L2A_T32TPS_D_V3-1/

After S2_prepare.py, data directory: workspace/decloud/data/S1_PREPARE/T32TPS/

After the python meraner_processor.py, This is following error while using the merunet_occitanie_pretrained model;

root@3dfcdc1d2ec0:/workspace/decloud/decloud/production# python meraner_processor.py --in_s2 workspace/decloud/data/S2_PREPARE/T32TPS/SENTINEL2A_20230821-101805-444_L2A_T32TPS_D_V3-1/ --il_s1 workspace/decloud/data/S1_PREPARE/T32TPS/ --in_dem workspace/decloud/data/DEM_PREPARE/T32TPS.tif --output workspace/decloud/data/output/ --savedmodel workspace/decloud/data/model/merunet_occitanie_pretrained/ 2024-07-15 13:35:23 (INFO) [pyOTB] Successfully loaded 126 OTB applications
2024-07-15 13:35:23 INFO Init. S2_TILED product
2024-07-15 13:35:23 INFO Product is not of type s2 (Exception: Not a S2_TILED product (expected 1 files ending with _FRE_10m.tif))
2024-07-15 13:35:23 INFO Init. S2_THEIA product
2024-07-15 13:35:23 INFO Input type is a directory
2024-07-15 13:35:23 INFO Product is not of type s2 (Exception: workspace/decloud/data/S2_PREPARE/T32TPS/ not a S2_THEIA product : _FRE_B2.tif is missing)
2024-07-15 13:35:23 INFO Init. S2_ESA product
2024-07-15 13:35:23 INFO Input type is a directory
2024-07-15 13:35:23 INFO Product is not of type s2 (Exception: workspace/decloud/data/S2_PREPARE/T32TPS/ not a S2_ESA product : _B02_10m.jp2 is missing)
2024-07-15 13:35:23 INFO Init. S1_TILED product
2024-07-15 13:35:23 INFO Product is not of type s1 (Exception: Not a S1_TILED product. File workspace/decloud/data/S1_PREPARE/T32TPS/ not found)
2024-07-15 13:35:23 INFO Product is not of type s1 (Exception: S1_THEIA.init() got an unexpected keyword argument 'verbose')
Traceback (most recent call last):
File "/workspace/decloud/decloud/production/meraner_processor.py", line 183, in
sys.exit(main())
File "/workspace/decloud/decloud/production/meraner_processor.py", line 177, in main
meraner_processor(params.il_s1, params.in_s2, params.savedmodel, params.in_dem,
File "/workspace/decloud/decloud/production/meraner_processor.py", line 68, in meraner_processor
input_s1_products.sort(key=_closest_date, reverse=True)
File "/workspace/decloud/decloud/production/meraner_processor.py", line 66, in _closest_date
return abs(s2t_product.get_timestamp() - x.get_timestamp())
AttributeError: 'NoneType' object has no attribute 'get_timestamp'

Can you please verify the pre-processed files and why I am facing this error? Did you update the pre-processing scripts through which we have different filenames saved in S1_PREPARE and S2_PREPARE and the pre-trained model is expecting different filenames?

[remicres]

Hi @abhisheksinghIN ,

I think that you must provide the absolute paths. Let me know if this fixes the issue

[BernradMaillard]

After S1_prepare.py; data directory: workspace/decloud/data/S2_PREPARE/T32TPS/SENTINEL2A_20230821-101805-444_L2A_T32TPS_D_V3-1/

After S2_prepare.py, data directory: workspace/decloud/data/S1_PREPARE/T32TPS/

After the python meraner_processor.py, This is following error while using the merunet_occitanie_pretrained model;

root@3dfcdc1d2ec0:/workspace/decloud/decloud/production# python meraner_processor.py --in_s2 workspace/decloud/data/S2_PREPARE/T32TPS/SENTINEL2A_20230821-101805-444_L2A_T32TPS_D_V3-1/ --il_s1 workspace/decloud/data/S1_PREPARE/T32TPS/ --in_dem workspace/decloud/data/DEM_PREPARE/T32TPS.tif --output workspace/decloud/data/output/ --savedmodel workspace/decloud/data/model/merunet_occitanie_pretrained/ 2024-07-15 13:35:23 (INFO) [pyOTB] Successfully loaded 126 OTB applications 2024-07-15 13:35:23 INFO Init. S2_TILED product 2024-07-15 13:35:23 INFO Product is not of type s2 (Exception: Not a S2_TILED product (expected 1 files ending with _FRE_10m.tif)) 2024-07-15 13:35:23 INFO Init. S2_THEIA product 2024-07-15 13:35:23 INFO Input type is a directory 2024-07-15 13:35:23 INFO Product is not of type s2 (Exception: workspace/decloud/data/S2_PREPARE/T32TPS/ not a S2_THEIA product : _FRE_B2.tif is missing) 2024-07-15 13:35:23 INFO Init. S2_ESA product 2024-07-15 13:35:23 INFO Input type is a directory 2024-07-15 13:35:23 INFO Product is not of type s2 (Exception: workspace/decloud/data/S2_PREPARE/T32TPS/ not a S2_ESA product : _B02_10m.jp2 is missing) 2024-07-15 13:35:23 INFO Init. S1_TILED product 2024-07-15 13:35:23 INFO Product is not of type s1 (Exception: Not a S1_TILED product. File workspace/decloud/data/S1_PREPARE/T32TPS/ not found) 2024-07-15 13:35:23 INFO Product is not of type s1 (Exception: S1_THEIA.init() got an unexpected keyword argument 'verbose') Traceback (most recent call last): File "/workspace/decloud/decloud/production/meraner_processor.py", line 183, in sys.exit(main()) File "/workspace/decloud/decloud/production/meraner_processor.py", line 177, in main meraner_processor(params.il_s1, params.in_s2, params.savedmodel, params.in_dem, File "/workspace/decloud/decloud/production/meraner_processor.py", line 68, in meraner_processor input_s1_products.sort(key=_closest_date, reverse=True) File "/workspace/decloud/decloud/production/meraner_processor.py", line 66, in _closest_date return abs(s2t_product.get_timestamp() - x.get_timestamp()) AttributeError: 'NoneType' object has no attribute 'get_timestamp'

Can you please verify the pre-processed files and why I am facing this error? Did you update the pre-processing scripts through which we have different filenames saved in S1_PREPARE and S2_PREPARE and the pre-trained model is expecting different filenames?

provide the relative paths, this worked for me.

 meraner_processor \
  --il_s1 ./images/S1/S1_PREPARE/s1a_33PUK_vvvh_ASC_161_20231029txxxxxx_from-10to3dB.tif \
  --in_s2 ./images/S2/S2_PREPARE/T33PUK/SENTINEL2B_20231030-093821-936_L2A_T33PUK_C_V3-1/ \
  --in_dem ./images/T33PUK.tif \
  --output reconstructed.tif \
  --savedmodel ./merunet_burkina_pretrained \
  --pad 256