GUI based Photorealistic Condor Scenery Tutorial - draft

Everything related to creation of new sceneries for Condor...

Moderators: Uros, OXO, GR, ASH33M

mtecles
Posts: 11
Joined: Sun Jan 17, 2016 10:53 pm

Re: GUI based Photorealistic Condor Scenery Tutorial - draft

Postby mtecles » Sat Jan 23, 2016 7:31 pm

Thank you, Tom, for your answer.

I decided to start with a small scenery to find out the procedures and problems to bild a scenery. Still it covers more than the area that we could fly there (the air space have just changed). Once I succeed I will try to bild a biger scenery for Bebedouro (SDBB), Sao Paulo state, a vary good place to fly.

I have both .94 and .95 of the spreadsheet. In both I can't find where to input the name of the csv file name (CSVFileName) that would be used on the gdal command to cut the tiles (Terminal Entries, D18). Both on User_inputs (D46-D50)and Terminal Entries (E6-E11) tabs I see just folder names inputs.

I also suspect of the iamges projection. They are WRS- 2L andsat 7 ETM+ Level 1 GeoTIFF files from EarthExplorer. One folder of then, for example, is LE72180762015043CUB00, with 9 tif images, 3 text files (LE72180762015043CUB00_MTL.txt, LE72180762015043CUB00_GCP.txt, README.GTF) and one gap_mask folder. In the LE72180762015043CUB00_MTL.txt I can read:

GROUP = PROJECTION_PARAMETERS
MAP_PROJECTION = "UTM"
DATUM = "WGS84"
ELLIPSOID = "WGS84"
UTM_ZONE = 23
GRID_CELL_SIZE_PANCHROMATIC = 15.00
GRID_CELL_SIZE_REFLECTIVE = 30.00
GRID_CELL_SIZE_THERMAL = 30.00
ORIENTATION = "NORTH_UP"
RESAMPLING_OPTION = "CUBIC_CONVOLUTION"
SCAN_GAP_INTERPOLATION = 2.0
END_GROUP = PROJECTION_PARAMETERS

Thank you,
Mauricio

tberry wrote:Mauricio - I've been updating the spreadsheet to try to simplify the process and that is what is causing the confusion.

First - I think that 4 X 4 is the minimum size for Condor!
Second, do you have version .95 of the spreadsheet? Look at the "Info" tab to see. If so, you can input the values manually on the terminal entries tab in cells E6-E11 on that sheet.
Third - you are correct about the missing space in the sheet for the terminal entry to cut the tiles! Thank you for pointing it out. I'd changed this section but didn't test it. That was a bad idea. I'm glad you figured it out.

Lastly, do you know the details of the source images you are using to make the scenery? The black output is probably because the projection of the tiles is different from the U.S. ones. I would like to help you figure this out so other non-US image sources can be used.

User avatar
jbreznai
Posts: 523
Joined: Wed Mar 29, 2006 2:21 pm
Location: Hungary / Budapest

Re: GUI based Photorealistic Condor Scenery Tutorial - draft

Postby jbreznai » Mon Jan 25, 2016 11:52 am

Hi!
First.... Excuse me for my english language, because I started learning when I was 50. Now I am 66... :D
Please help me !
I try this process, and I calculated in .xlsx my developed scenery, in CST is ready, in Condor is flyable, but have not got texture... (17x19 tiles 4864x4352)
I tryed make in QGIS (2.12. Lion) point matrix , it managed but the number of tiles are not correct.. I think the mistek is the number of 0.2133 ...... wich depend on the pixel size (E24 > 0,0008334 ) . What is the correct number, I not finde the info in QGIS (<---Critical to accuracy! Use QGIS Raster->Misc->Info to confirm).

Thanks
Janos
[url=http://condor.ephemeride.com/graduate/0/?id=297]
Image[/url]
Image

tberry
Posts: 91
Joined: Thu Aug 09, 2012 4:10 pm

Re: GUI based Photorealistic Condor Scenery Tutorial - draft

Postby tberry » Mon Jan 25, 2016 5:38 pm

Janos, are you using the Google Spreadsheet version .95? Early on, I had an Excel equivalent to the Google sheet but discontinued it. There are just a enough incompatibilities between the two that each time I updated the spreadsheet I had to review both to debug. Until I'm sure I have a stable version, the latest is the Google Sheet that the first post points to.

Also, you are following the tutorial steps 1.2B, also?

For your terrain, I assume you are using the SRTM3 data, correct? If so, 0.2133 is the correct spacing based on a pixel size in the SRTM data of .0008334. You can confirm this in QGIS as follows:

1. Open QGIS and use Project->New to be sure you have a clean project
2. In the lower right corner of the QGIS window, click once on the CRS to change the EPSG setting (next to the "Render" checkbox) and search for and use EPSG:4326.
3. Click OK and be sure the CRS now says "EPSG:4326" Be sure the "Render" checkbox is checked.
4. Drag and drop one of your SRTM tiles into the main window. It should display the tile.
5. In the QGIS menus, select "Raster->Miscellaneous-Information" and use the "Input file" dropdown to select your SRTM tile layer. Click OK.

In the Raster info window of that dialog, you will see information about the SRTM tile. Scroll down until you see "Pixel Size". It should be 0.000833333 repeating.

If it is, then the spreadhsheet numbers are correct. I notice that you reversed your reference in your note. Perhaps it was just a mis-type but 17X19 tiles is not 4864X4352. It is the reverse - 4352X4864. This is for the terrain mesh, of course. The number of pixels for the images will be different, depending on the source images you use. In fact, I am looking at your landsat post to try and help someone use imagery for a scenery in South America!

Hope this helps.

tberry
Posts: 91
Joined: Thu Aug 09, 2012 4:10 pm

Re: GUI based Photorealistic Condor Scenery Tutorial - draft

Postby tberry » Mon Jan 25, 2016 5:56 pm

I have updated the versions of the tutorial and spreadsheet that are linked in the very first post. Although I am not quite finished with the re-draft of the tutorial, I've been updating both the spreadsheet and tutorial based on inputs from people who have tried them out.

Creating the spreadsheet as a Google template is great because any changes I make automatically update the template. But, I've changed the spreadsheet enough that people are seeing the 'new' version of the spreadsheet and trying to follow the 'old' version of the tutorial. They don't match very well, anymore!

So, I've put the draft .95 tutorial out there so it more closely matches the spreadsheet .95. Embarrassingly, I'm finding some basic and stupid errors I've introduced into the spreadsheet that stops some people cold -mainly in the Terminal Entries tab where I've done more refactoring of the sheet to make it simpler to use. These are very small, but signficant errors like not putting a space between the source and destination files so that the terminal entry makes an error.

Keep posting any errors you find and thanks for trying out my process. I'm getting better at troubleshooting and I'm working a process for importing Landsat images so you can use scenery sources from anywhere in the world.

User avatar
jbreznai
Posts: 523
Joined: Wed Mar 29, 2006 2:21 pm
Location: Hungary / Budapest

Re: GUI based Photorealistic Condor Scenery Tutorial - draft

Postby jbreznai » Mon Jan 25, 2016 9:17 pm

Dear Tom!

I tryed both .xlsx files.

I received in both case same problem. When I opened files Windows try recower .xlsx but not managed :(
(My Office sysem is 64 bit, My PC: i7 Intel 3,2GB ; 8GB Ram ; 1TB HDD)

Xml file, of error

<?xml version="1.0" encoding="UTF-8" standalone="true"?>
-<recoveryLog xmlns="http://schemas.openxmlformats.org/spreadsheetml/2006/main">
<logFileName>error051280_01.xml</logFileName>
<summary>Hiba a fájlban (E:\CODOR FEJLESZTÉS\Photorealistic scenery\Condor Scenery Spreadsheet.xlsx)</summary>
-<removedRecords summary="Az eltávolított rekordok listája:">
<removedRecord>Eltávolított rekordok: Képlet innen: /xl/worksheets/sheet1.xml</removedRecord>
<removedRecord>Eltávolított rekordok: Képlet innen: /xl/worksheets/sheet5.xml</removedRecord>
</removedRecords>
</recoveryLog>

For QGIS I received information from .dem file of scenery wich I generated in 3DEM ( save as the .bin file)
OGIS I would like to use only for textures process, and the GRASS for phansharpening ...
SRTM files are OK, the scenery working, flyabel.
I going to try again the pointmatrix and I generate new shape file..

Other question... I used for my sceneries Landsat images ... Now I working with Landsat 8... but Landsat pictures has only 15m/pixel resolution, and I dont known better source for high resolution pictures (than AA scenery)

Thanks your answer
Br
Janos
[url=http://condor.ephemeride.com/graduate/0/?id=297]

Image[/url]

Image

mtecles
Posts: 11
Joined: Sun Jan 17, 2016 10:53 pm

Re: GUI based Photorealistic Condor Scenery Tutorial - draft

Postby mtecles » Mon Jan 25, 2016 10:52 pm

Hi from Brazil, Janos!

I have tried to download and open the spreadsheet in my computer too (Windows/Office). I had the same error, so I use the spreadsheet online. Just login, select the spreadsheet (click on it), you will have your own copy. That is what I am doing.

I am having problems to process the images and I am trying to figure it out. I will try with the new Tutorail (.95) as soon as I can. Thank you, Tom!

Mauricio

jbreznai wrote:Dear Tom!

I tryed both .xlsx files.

I received in both case same problem. When I opened files Windows try recower .xlsx but not managed :(
(My Office sysem is 64 bit, My PC: i7 Intel 3,2GB ; 8GB Ram ; 1TB HDD)

Xml file, of error

<?xml version="1.0" encoding="UTF-8" standalone="true"?>
-<recoveryLog xmlns="http://schemas.openxmlformats.org/spreadsheetml/2006/main">
<logFileName>error051280_01.xml</logFileName>
<summary>Hiba a fájlban (E:\CODOR FEJLESZTÉS\Photorealistic scenery\Condor Scenery Spreadsheet.xlsx)</summary>
-<removedRecords summary="Az eltávolított rekordok listája:">
<removedRecord>Eltávolított rekordok: Képlet innen: /xl/worksheets/sheet1.xml</removedRecord>
<removedRecord>Eltávolított rekordok: Képlet innen: /xl/worksheets/sheet5.xml</removedRecord>
</removedRecords>
</recoveryLog>

For QGIS I received information from .dem file of scenery wich I generated in 3DEM ( save as the .bin file)
OGIS I would like to use only for textures process, and the GRASS for phansharpening ...
SRTM files are OK, the scenery working, flyabel.
I going to try again the pointmatrix and I generate new shape file..

Other question... I used for my sceneries Landsat images ... Now I working with Landsat 8... but Landsat pictures has only 15m/pixel resolution, and I dont known better source for high resolution pictures (than AA scenery)

Thanks your answer
Br
Janos

tberry
Posts: 91
Joined: Thu Aug 09, 2012 4:10 pm

Re: GUI based Photorealistic Condor Scenery Tutorial - draft

Postby tberry » Mon Jan 25, 2016 11:14 pm

Okay, I see. You already have a flyable terrain. That means you already know the corners for the scenery lat/lon, correct?

If so, you can simply use the terminal commands in the spreadsheet to process the landsat images after you are finished with processing them.

But there is no .xlsx sheet. Just run it in the browser as a Google spreadsheet. It works very well.

There are a few things that will likely come up:

1. It may actually be easier to use landsat images than the naip .jp2 images in the tutorial. This is because landsat images are delivered as GEOTIFFS and are in WGS84 by zone. I just downloaded a test image and reprojected it to EPSG 4326, which is Condor format. So, that's good news! It may require fewer steps, but they will be different than the ones in the tutorial.

2. Because of the different zones, I don't know if they can all be combined in one step. I'll have to download a few more to figure that out.

3. I need to understand how to combine the various bands to get a natural color image. Right now, the one I have is just black and white.

I think the way for you to proceed is to make all adjustments to the GEOTIFF files first and output a set of "source" files for your scenery. Then, we assemble the source images as vrt files, reproject them as vrt, cut them to size as a vrt, then cut the tiles from that vrt and output as bmp.

User avatar
jbreznai
Posts: 523
Joined: Wed Mar 29, 2006 2:21 pm
Location: Hungary / Budapest

Re: GUI based Photorealistic Condor Scenery Tutorial - draft

Postby jbreznai » Tue Jan 26, 2016 10:39 am

Hi Tom!

Okay, I see. You already have a flyable terrain. That means you already know the corners for the scenery lat/lon, correct?
I always make . dem file in 3DEM. After I open .dem file in MICRODEM (http://www.usna.edu/Users/oceano/pguth/ ... crodem.htm) , from here I received the correct center and corners info. In Microdem I add to .dem of scenery shape files (road, water, lake.. etc... ) and I save as than .bmp file. It will be the flyplanner map, for navigation device (PDA)
If so, you can simply use the terminal commands in the spreadsheet to process the landsat images after you are finished with processing them.

But there is no .xlsx sheet. Just run it in the browser as a Google spreadsheet. It works very well.
I going to try it.

There are a few things that will likely come up:

1. It may actually be easier to use landsat images than the naip .jp2 images in the tutorial. This is because landsat images are delivered as GEOTIFFS and are in WGS84 by zone. I just downloaded a test image and reprojected it to EPSG 4326, which is Condor format. So, that's good news! It may require fewer steps, but they will be different than the ones in the tutorial.
I never used naip .jp2 images :(
2. Because of the different zones, I don't know if they can all be combined in one step. I'll have to download a few more to figure that out.

3. I need to understand how to combine the various bands to get a natural color image. Right now, the one I have is just black and white.
On first time I used FVTools... and on next time I found "High view" for Landsat 7 images, but this program could not handle Lantsat 8 images... but Geosage (http://www.geosage.com/) developed in 2013 Spectral Transformer for Landsat-8 imagery it was free, but now is not free :( . I used for my Hungary_2 scenery wich based on Landsat 8 images.
When I going to finish I would like start with my new scenery ( South-East part of Poland name is "Podkarpackie") because was a litle trip with QGIS and Grass > Grass able to handle fusion and pansharpening process in one step. http://courses.neteler.org/processing-l ... harpening/



I think the way for you to proceed is to make all adjustments to the GEOTIFF files first and output a set of "source" files for your scenery. Then, we assemble the source images as vrt files, reproject them as vrt, cut them to size as a vrt, then cut the tiles from that vrt and output as bmp.
Good idea... :)
When I received the big GEOTIFF file (size over 6-7 GB) I could open only in FW tools and Image linker and QGIS. I converted to .psb format and I cam open in Photoshop ( size only 2-3 GB for 2048x2048) wich can handle the Photoshop script. Because very good your idea. :)


BR
Janos
[url=http://condor.ephemeride.com/graduate/0/?id=297]

Image[/url]

Image

tberry
Posts: 91
Joined: Thu Aug 09, 2012 4:10 pm

Re: GUI based Photorealistic Condor Scenery Tutorial - draft

Postby tberry » Tue Jan 26, 2016 6:50 pm

Janos, last night I found a plug-in for QGIS that I think will do MUCH of what you want with Landsat data. It is called Semi-automatic classfication plug-in. It is amazing and it is free. Watch the video of features here: http://www.digital-geography.com/landsat-in-love-with-qgis-the-newest-coup-from-luca/#.VqarH1MrJTY

It is complex, but I think you need only a few of the features. There are tutorials here: http://fromgistors.blogspot.com/p/theinterface-2.html?spref=scp

Basically, you can define the area you want Landsat 4,5,6,7,8 data for, and this tool will 1) find it, 2) allow you to preview it in QGIS, 3) Download the selected images, 4) Perform automatic pan-sharpening and other pre-processing steps.

I'm sure it does more, but as I said, it is a bit complex and will require some reading and experimentation to figure out the few exact steps needed to import data.

You can add the plug-in on the QGIS plugin menu by searching for it in the repository and installing it.

Also, QGIS should be able to display multiple images from landsat data. It is pretty good at making summary tiles for viewing on the screen. You can get a lot of tiles on there before you start bogging it down.

User avatar
jbreznai
Posts: 523
Joined: Wed Mar 29, 2006 2:21 pm
Location: Hungary / Budapest

Re: GUI based Photorealistic Condor Scenery Tutorial - draft

Postby jbreznai » Tue Jan 26, 2016 10:11 pm

Hi Tom!

I made 3 scenery with Landsat images... My openien that the Landsat images have to downlad manually from USGS site. For one row/column group contain ~50 pc pictures with different preparation time, and from 50 images are without clauds only 1 or 2 pc.

My expeience that for one scenery have to use max. 20 Landsat tiles.
Have to pansharpening these ... it is very fast process with Geosage or Grass method, also manually.

From pansharpened images have to make mosaic file, and from mosaic file have to cat the big geoTIGFF (GDAL_GTiff) file and from geoTiff have to make .bmp tiles... For this process recommended the QGIS, so far I used Imageliker. With Imagelinker the process very easy if we want only low resolution scenery. (Size limit, GDAL_GTiff to .psb > photoshop) If we can receive high resolution satellit images (target) 5pixel/m (4096x4096 tile) or higher the GEOTIFF will be 4x or 16x bigger. We can handle these only in QGIS.

Br
Janos
[url=http://condor.ephemeride.com/graduate/0/?id=297]

Image[/url]

Image

tberry
Posts: 91
Joined: Thu Aug 09, 2012 4:10 pm

Re: GUI based Photorealistic Condor Scenery Tutorial - draft

Postby tberry » Tue Jan 26, 2016 11:08 pm

Thanks for the information, it sounds like you know a lot about Landsat images!

The NAIP images are 1 meter resolution. You must download several thousand of them -about 60gb for the source images. Working with this many files and that much data has been challenging, but QGIS and gdal are able to handle them using .vrt virtual raster files. Are you using these? They make the processing SO much faster and do not require you to build massive GEOTIFF files.

The gdal_retile command is able to cut a virtual raster file into the exact sized Terragen tiles that are GEOTIFFS. Converting them into .bmp files is simple after that. I think it is possible to make it output DDS tiles, but haven't explored it enough. In any case, LE takes care of that easily.

mtecles
Posts: 11
Joined: Sun Jan 17, 2016 10:53 pm

Re: GUI based Photorealistic Condor Scenery Tutorial - draft

Postby mtecles » Wed Jan 27, 2016 12:19 am

Hi, Tom!

For your information, in the spreadsheet .95 (January 2016) it is still missing the input cell for CSVFileName used on the gdal_retile command (terminal Entries, D18, Cut the tiles; 4.2.A, page 103, there is no name after the -csv option, 2016 Tutorial). There are no User_inputs D51 and D53 cells (Appendix F, pages 150 and 152). Of course, I can copy the command and insert a name after -csv and apply it to the gdal command shell.

Wow! I have to say again, your tutorial and spreadsheet are a pieces of art, and getting even better! I barely can´t wait to try it again.

Thank you!
Mauricio

tberry wrote:I have updated the versions of the tutorial and spreadsheet that are linked in the very first post. Although I am not quite finished with the re-draft of the tutorial, I've been updating both the spreadsheet and tutorial based on inputs from people who have tried them out.

Creating the spreadsheet as a Google template is great because any changes I make automatically update the template. But, I've changed the spreadsheet enough that people are seeing the 'new' version of the spreadsheet and trying to follow the 'old' version of the tutorial. They don't match very well, anymore!

So, I've put the draft .95 tutorial out there so it more closely matches the spreadsheet .95. Embarrassingly, I'm finding some basic and stupid errors I've introduced into the spreadsheet that stops some people cold -mainly in the Terminal Entries tab where I've done more refactoring of the sheet to make it simpler to use. These are very small, but signficant errors like not putting a space between the source and destination files so that the terminal entry makes an error.

Keep posting any errors you find and thanks for trying out my process. I'm getting better at troubleshooting and I'm working a process for importing Landsat images so you can use scenery sources from anywhere in the world.

tberry
Posts: 91
Joined: Thu Aug 09, 2012 4:10 pm

Re: GUI based Photorealistic Condor Scenery Tutorial - draft

Postby tberry » Wed Jan 27, 2016 2:13 am

Thanks, Mauricio. I'm glad you haven't been thrown by the changes.

I'm fixing the spreadsheet right now and have made it a bit of a mess as I simplify it. I've reworked it to move those entries to a different tab and separated some stuff out. It's CLOSE to being finished. I'm sorting out the process for cutting a partial rectangle right now, then I'll put that in the manual and call it version 1.0.

-Tom

tberry
Posts: 91
Joined: Thu Aug 09, 2012 4:10 pm

Re: GUI based Photorealistic Condor Scenery Tutorial - draft

Postby tberry » Thu Jan 28, 2016 7:18 pm

Version .96 of both the tutorial and spreadsheet are updated. Links are in the very first post of this thread.

This version is VERY close to the final. The spreadsheet has been updated and simplified considerably and the tutorial rewritten to follow. I expect any updates I make to both will be to correct errors. There is still a bit more to go in section 6:
    -Where to post your sceneries when finished
    -How to use Sketchup to add buidlings and objects
    -Techniques for color balancing tiles from different image series
    -Tile cutting troubleshooting

I'm posting this because even in its unfinished state, this version is a significant improvement over the previous tutorial/spreadsheet combination.

If I figure out the landsat source instructions, I'll also include them to make the process applicable to worldwide sites and be less US centric.

pstrzel
Posts: 264
Joined: Sat Sep 08, 2012 2:02 am
Location: Utah

Re: GUI based Photorealistic Condor Scenery Tutorial - draft

Postby pstrzel » Fri Jan 29, 2016 1:36 am

tberry wrote:Thanks for the information, it sounds like you know a lot about Landsat images!

The NAIP images are 1 meter resolution. You must download several thousand of them -about 60gb for the source images. Working with this many files and that much data has been challenging, but QGIS and gdal are able to handle them using .vrt virtual raster files. Are you using these? They make the processing SO much faster and do not require you to build massive GEOTIFF files.

The gdal_retile command is able to cut a virtual raster file into the exact sized Terragen tiles that are GEOTIFFS. Converting them into .bmp files is simple after that. I think it is possible to make it output DDS tiles, but haven't explored it enough. In any case, LE takes care of that easily.


Tom,
If you order from https://gdg.sc.egov.usda.gov/GDGOrder.aspx you can download entire counties in MrSid format (15:1 compression), open it in LizardTech's free Geoviewer, "bookmark" your scenery boundries and save to GeoTIFF. The paid Geoviewer has scenery tiling features as well. I found ver. 9 to be a little buggy, so I've been using 5.5, which doesn't have the bookmark feature, and then using gdal tools to cut to size.

Obviously I've gotten a little off-track since this thread is about QGIS, but my point is that you don't necessarily need to have "several thousand" downloads. Have you explored this avenue in conjunction with QGIS?
Image


Return to “Scenery”

Who is online

Users browsing this forum: No registered users and 4 guests

cron