CDROM.txt 6.7    HOW TO BUILD APRS MAPS FROM THE USGS CD ROM      25 Mar 95

New in 6.7:  Brought README.100 into this file to reduce the # of txt files

NOTE:  Special guidance for mapmakers is in README\MOREMAPS.txt

WARNING:  Trying to make an APRS map from the 100,000:1 USGS CD ROMS should
be considered EXPERIMENTAL and quite TIME CONSUMING at this point.  Do not
consider doing it, unless you accept a challenge or think you can help
us find a quicker way to do it.  For now, it is more fun to me to use
a DIGITIZER (if you have a paper map) and take a full evening, than the
time involved in the multiple passes with the tools that follow... If
however, your digitizer is only 11 inches square and you have to re-place
the MAP 4 or more times to get it all in, then I would say it is a toss-up.

BEGIN:  All information on using the original 2,000,000:1 USGS CD ROM has
been moved to the end of this file, since by now, (fall 94) almost all
of the US population centers have been made into APRS maps at this level of
detail.  This FILE now begins to discuss the use of 100,000:1 USGS CD ROMS
that contain data on roads all the way down to your street.  Because of the
tremendous amount of data available in this data base, USGS sells the data on
a set of 13 CD's.  To see which CD covers your area, hit 1 on your APRS screen
to get to the whole USA map.  Then press MAPS-ALT command to see the CDROM.map
file which is under the USA map.  This display shows you the areas of the 13
different ROMS.  My yellow lines are crude, but the data in the files does
truncate exactly at the state lines.  To obtain one of these CD ROMS, send $32
for each ROM to the USGS Earth Science Info Center, 507 National Center,
Reston, VA 22092. 703-648-6045 or call 1-800-USA-MAPS.
BE SURE TO SPECIFY THE OPTIONAL FORMAT!!!!!!!!!!!!!!!! (vice standard)

WARNING:  Since both Ken, W7KKE and myself are continuing to improve these
programs, and keep modifying each others code, the following programs are
identical, except for the latest mods installed by each writer.  Look at the
latest file date, to see which one is the most recent:

     Ken's:   CONV100.bas   =   Bob's 100toGRF.bas  (Currently, use KEN's)
              DEFRAG.bas    =         JOIN.bas  (currrently Ken's is better)

JUST BECAUSE WE HAVE NOW WRITTEN ROUTINES TO HELP YOU MAKE MAPS FROM BOTH THE
2,000,000 and 100,000 SCALE USGS CD ROMs, DO NOT GIVE UP ON MAKING YOUR OWN
MAPS USING ANY OF THE OTHER METHODS.  THE 2,000,000 SCALE CD IS ONLY GOOD FOR
64 MILE MAPS because THEY DONT HAVE ENOUGH DETAIL TO MAKE SMALLER MAPS, and
they HAVE TOO MANY POINTS TO MAKE MAPS MUCH LARGER!  The 100,000 scale maps
generate so many points that an APRS map of city streets can be no larger than
about 1 mile in dense neighborhoods!

BACKGROUND:  There are 3 problems with the USGS data that causes excessive
points in an APRS map.  First, lines begin and end at EACH intersection.  For
am interstate, for example, you end up with dozens of short road segments
wasting 3 point pairs per exit.  Second, these short segments are not always
consecutive in the file, making it extremely time consuming to try to join
them.  Third, the USGS data guarantees a point every .01 inch at the original
maps scale (1/3 mile for the 64 mile maps) whether it is needed or not.  A 10
mile long straight road is stored as 30 points in USGS vice 2 points in APRS.
At the 100,000:1 scale, .01 inch is only 100 feet!


VERY FINE DETAIL USING THE USGS 100,000:1 DETAIL CD ROMS

    W7KKE wrote a program called CONV100 which takes data from the 100,000
scale CD ROMS.   It generates so many
roads in an urban environment, that just a 1 square mile map has over 5000
points.  I have upped the APRS maximum limits to accomodate up to 2999 points
but you should avoid making these large size maps unless you absolutely
need every street, because the large size also means APRS takes longer to
load maps!  Since we are vehicle tracking for radio purposes, a whole map
1 mile square is only one one-thousandth of the potential coverage area of
one APRS digipeater, so you can see that the effort to make these maps is
not worth the time in my opinion. (unless you are making the detail for
a commercial application and are willing to spend the time!)  Here is how
to do it:

1.  Run the USGS EXTRACT program and follow its instructions to output the
appropriate map data for the 15 x 15 minute area that you are interested in.
These files will be of the form XXXRDSnn for ROADS and xxxHYFnn for hydro
where the nn is for multiple files to cover the area.  In my urban area,
each of the ROAD files covers about a 2 mile area.

2.  Run CONV100.bas and once you give it one of the above file names, it will
automatically generate a series of 10 output files of the form PREnRD.GRF for
n different ROAD files and two other files preWB.grf and preST.grf for
waterbodies and streams.  100toGRF.bas outputs these files in the same format
as the original 2,000,000 that MAPFIX knows how to read.  The PRE is a 3 or 4
letter prefix that you specify for this series to keep it separate from other
efforts.  So far, the number 4, 9 and 10 maps in my area have been empty...
but the pre8RD.GRF file is the largest by a factor of ten!  It contains all
of the neighborhood streets.

3.  Run MAPFIX and load an existing APRS map that covers the area where
you will be working.  This map just makes it easy to visually get the
cursor to the right area.  Once you hit the ALT-U commmand, this existing
MAP is no longer used.  MOve the cursor to the center of the map you want
to make, and hit the B command to establish the boundary of the area
that you want to develop.  Choose a scale of 2400, 3000, 3600, 4800 or so
that will cover the total area you eventually want covered.  For city
streets, you should use at least 2400 or more.  Next use the alt-USGS
command to bring in the first of the above files.  Then do the alt-SMOOTH
and the alt-JOIN commands to minimize the map.  Then SAVE IT WITH A NEW
NAME.  Then repeat this step for all the maps that were output from the
CONV100.bas program.  After you have saved each of these, you will now
have almost a dozen maps each containing a different feature, but in APRS
format. 

WARNING!:  NEVER do a second ALT-U USGS BUILD on the same map, because it
will clear all current points and start a new map from scratch.

4.  Next, run Ken's DEFRAG.bas program on EACH of the files generated by
MAPFIX in the above step 3.  This program goes through the files over and
over again, trying to join together all of the disjointed road segments
(of the same class) generated by the USGS data base.  Hopefully you may
see a 20% or more reduction in points.  Re-save each of these reduced
maps under a new name so that you do not distroy your originals (if there
is a problem.

NOTE:  Be sure to also review the steps at the end of this file on how MAPFIX
is used to read the 2,000,000:1 CD ROMS.  Since the 2,000,000:1 process was
developed first and is the basis for MAPFIX routines, you might find some
helpful details or clarifying information.  Ken Swaggart, W7KKE, wrote the
conversion routines to put the 100,000:1 OPTIONAL data format into the
GRAPHICS format available on the 2,000,000:1 CD ROM.

His instructions are in README.100, NOW INCLUDED AT THE END OF THIS FILE.

4.  Now, combine these maps into one APRS map, using the MAPFIX alt-IMPORT
command.  This command will bring just one, or all of the map features of one
file into the current file.  In an urban area, the level 8 road file will be
larger than all the others combined.  You can omit it to allow your map to
cover a larger area or include it for the finest detail available.

Good luck.  I am just beginnning to play with this level of detail, and to
this point, it is far more practical to use a digitizer!  For more detail on
the use of MAPFIX with CD ROMs, also read the 2,000,000:1 section below,
since it evolved and was the basis for APRS maps before we began messing with
the 100,000:1 data.



PROCEDURES FOR 2,000,000:1 CD ROM THAT COVERS THE WHOLE US

I can do about one map in a full evening, and most of the USA is now done.
If you need a 64 mile map of your area that is not already in APRS, lemme
know and I might just do one...

1.  Use USGS XTRACT.EXE program to extract the CD ROM data in the GRAPHIC
format for your area from one of the dozen or so major USGS areas.  Then
select up to 6 files for extraction onto your HARD disk, one for each of
the data types, BOUNDARIES, ROADS, WATER BODIES, STREAMS, RAILROADS, and
HYPSOGRAPHY (continental divide).  These files will take about 5 MBytes.
To save HD space, and also to end up with the minimum of points, you will
probably only use the ROADS, WATERBODIES, and STREAMS files.  Note the
filename assigned by XTRACT.

2.  Run MAPFIX.BAS program and load the best existing APRS map that covers
the desired area.  This map is only used to help you locate the area of
interest.  Next, move the cursor to the center of interest and select
a SCALE using the BOUNDARY (B) command.  This command will show you the
maximum map possible for a given scale in Points/per degree.  I have found
600 to be about right.  450 in sparse states and 300 in UTAH and NEW Mex.
If you use any less, to get larger maps, you will end up with far too many
points.  I used 450 for the SouthCarolina map, and it took me 6 hours to
get the resulting 3000 points down below 1500!  In the East, I used 900
for a 35 mile map around Philidelphia.  Making maps any smaller is probably
not worth the effort, since there will be so few roads in it..

3.  When you have the B command set to the desired number of Pixels-per-degree,
press the alt-U key.  MAPFIX will allow you to round the ORIGIN values, to
reasonable nearly whole numbers or fractions of a degree so that later
manual operations with a map and ruler will have round numbers to deal
with.

4.  Next APRS asks you for the type of file (RD, ST, WB, etc) and then the
exact path to the XTRACTed USGS data.  This path should include the first
part ot the filename up to the (_) underline that separates the RD or WB
etc final two characters.  On my system it is usually D:\NEAST\S01_  and
then MAPFIX adds the RD.GRF on the end.  MAPFIX will then scan the entire
indicated file for all points within the selected border.  While converting
the points to APRS format, MAPFIX will also compare the slope of each new
point with the previous point and ignore the point if it has approximately
the same slope (within 1.2)  The total number of points will be from 2000
to 4000.

4.  Next MAPFIX has a SMOOTHER command which will run another slope comparison
process.  You can select the SMOOTHER factor.  A factor of 1 will only
throw out points that are exactly the same slope (few if any).  A factor of
1.2 (best) will throw out points whose slope differs by less than 1.2.
Bigger values will throw out  more points (usually including some necessary
ones).  I never use a value greater than 1.5.  To test the smoothing
algorithm, run the TEST SMOOTHER command which will mark every point that
would be eliminated with a red dot, and every point that will be kept with
a yellow dot; but without actually eliminating any points.  Look at the map
closely and see if you can accept the decisions of the SMOOTHER.  Select
your chosen factor and run SMOOTHER. (alt-S)

5.  Next use the alt-JOIN command.  This will cause MAPFIX to scan through
the map looking for consecutive map features that end and then start
at the same point.  These two features will be joined into one longer
continuous feature.  At this point, you can save the map, quit MAPFIX
and run DEFRAG.bas which extends this same philosophy by comparing
each feature to EVERY other feature (not just consecutive) still trying
to match end points of one feature with the start of another.  This
process can usually eliminate another couple of hundred points.  The
result is saved in MAPTEMP.map.  DEFRAG does not do LABELS, so if you
run DEFRAG on a map with labels, they will be lost.

6.  Now you have the smallest APRS map that can be done automatically.
Reload MAPFIX, reload the MAPTEMP.map (or other optional name) and Move
the MAP POINTER to the beginning FEATURE using the RESET command and
use the gray +/- keys to cycle through EVERY point in the file.  Use the
alt-D key to delete all unnecessary points.  Look for complete duplicates
of some lines too.  Both lanes of a 4-lane or interstate are in the file
but will be EXACTLY the same!  This should get you down to below 2000 or
so.

7.  Notice that you only have one category of features so far (usually roads).
Unless you require county lines I recommend that you ignore them.  The
USGS BOUNDARIES file contains even more points than the ROADS file!  AND
the SMOOTHER routine does not work well, because property lines are very
jagged, unlike smooth curves in roads.  The same thing goes for WATER
BODIES and STREAMS.

10. To allow you to add some detail from these other USGS files, MAPFIX has
the USGS OVERLAY feature which will overlay the USGS data on the MAPFIX
screen without doing anything.  Then you can use the NEW-FEATURE and ADD
commands to draw APRS features over the top of these temporary points.
The philosopy here is that 96% of amateur interest is on roads, and you
can sure burn up points on waterways and county lines.  For rivers, about
the only points I use are where the water goes under a bridge.  Most folks
have no idea where a river is out in the boonies (and could care less).
Of course, if you are making a map for boats with GPS, then start the
process with the WATERBODIES file and add just enough roads for them to
find the boat ramps..
    You will first need to overlay on the 64 mile scale to see where things
are, but then zoom into the 16 mile scale at least and do the overlay again
to get enough resolution for drawing a particular feature.  Usually, I
sketch in rivers at the 32 mile range, because I do NOT try to get every
king in a river or stream...  You can use the NEW, ADD and MOVE commands,
but any other commands will redraw the map and wipe-out the overlay.  I
have added the Ctrl-R command to toggle the map REDRAW process on and off. 
This may help you to add lines without destroying the USGS overlay.  You
can do the overlay again and again.

WARNING!:  NEVER do a second ALT-U USGS BUILD, because it will clear memory
and start from scratch.


CAUTION:  Do not expect too much from this CD ROM.  ALthough the 2,000,000:1
USGS database seems to have resolution down to 1/3 rd of a mile, remember
that that 1/3rd of a mile was actually 1/100th of an inch on the original
map!  Your 64 mile section was only a two inch square!!!!!  AND THE DATA
WAS PROBABLY DERIVED BY A HIGH SCHOOL SUMMER STUDENT using a mouse!

      I have found this data to be very crude, and although the maps lines
look smooth, you are getting about the same quality as if you were
looking at a state highway map through a magnifying glass!  If you want
good detail, use MAPFIX and a digitizer on a much better original map,
such as a 100,000:1 or even 24,000:1 (which is the standard USGS 7.5
minute map).  There you can get down to your driveway!

IT STILL TAKES AN EXCESSIVE AMOUNT OF MANUAL EFFORT TO REDUCE THESE MBYTES
OF USGS DATA DOWN TO A reasonably quick loading 15K APRS MAP!  IT TAKES ME 4
to 12 HOURS PER MAP!  By now (Fall 94) I have done most of the US to the
state highway level of detail.  IF YOU WANT ME TO DO A SPECIAL MAP OF YOUR
AREA, to the resolution in the 2,000,000:1 ROM, SEND $$$ or convince me it is
in the general best interest of all HAMS.  THAT ALONE SHOULD KEEP YOU
INTERESTED IN DOING YOUR OWN.  HI HI

FOR THE 100,000:1, I will only do these maps commercially, because the 3 days
worth of effort only results in a map that covers a few miles.  Only a few
HAMS will even live inside one of these small areas.  If you are a MAP-ology
expert, or an ace programmer and can help us figure out a way to better
automate the process, then we would love to make the whole process easier
for everyone.  Contact Ken Swaggart, W7KKE in Calif, or myself.

*****************************************************************************
README.100   (as of 2/13/94)

                   1:100,000 CD ROM Map Making

                            W7KKE
                         Ken Swaggart
               w7kke @ ka6eyh.#nocal.ca.usa

The 1:100,000 line segment maps contained on the USGS CD ROM uses
a different format than the 1:2,000,000 CD ROM maps. My first
utility, CONV100.BAS, converts the 1:100,000 map "Optional Format"
extracted files to the .GRF file format which can be read by the APRS
MAPFIX.BAS program. When you import these map files into MAPFIX you
will notice they are extremely fragmented into small line segments
which are not adjacent. The second utility, DEFRAG.BAS takes the saved
MAPFIX file and defragments it.

Here are the step by step instructions:

  1. Create a directory on your hard drive where you will be work
on your maps. Stick my programs in this directory and copy
MAPFIX.BAS from the APRS directory into this working map directory.
I use C:\APRSMAPS as my working map directory.

  2. Extract "Transportation" and "Hydrography"  for your area of
interest from the 1:100,000 CD ROM using the extraction program
which comes on the CD ROM. I zoom down to the 15' scale on the USGS
extraction program before I extract the map file. Copy these files
into your working map directory.

  3. Run CONV100.BAS.  The program will extract each feature, such
as Interstates, US Highways, etc., as separate files with the
filename extension of .GRF.

  4. Run MAPFIX.BAS. Import the first converted .GRF file  by using
the @U command in MAPFIX.BAS. After the file is imported you and on
your MAPFIX screen use the Join command (@J) to join adjacent line
segments for this feature. Save this feature as its own .MAP file
and repeat the process for each map feature you created.

  NOTE: Make sure you use the same origin and scale each time you
        import the .GRF file using the @U command.

  5. Run DEFRAG.BAS to de-fragment the .MAP files you have created.

     If DEFRAG.BAS crashes, you can recover most of your defragmented
     segments. Look for SCRATCH1.TMP and SCRATCH2.TMP. Rename the
     larger of these two and delete the remaining .TMP file.

  6. Use a text editor, such as DOS's EDIT, and examine your
defragmented .MAP files and tweak as you see fit. You may find that
some line segments consist of only one point - EDIT THOSE OUT

  7. Run MAPFIX.BAS again and import your defragmented .MAP file.
Check it over for any obvious errors, correct those, then run the
smoother function (@SMOOTH) to pare down the .MAP file.

  Note: I generally do not run smoother on the water boundaries and
streams, but hand edit to reduce the points. Another option which
works well is to use the U command to overlay the water boundaries
and streams, then manually draw in the rivers with the @new and @A
commands. This results if far fewer points than the automatic
method.

 8. Save your smoothed .MAP file and import your next feature and
repeat the process. (I only run the smoother on each separate
feature file so I can control what gets smoothed.) Sometimes I have
to bump the smoother factor way up to pare down the points to fit
the total point limit for APRS.

  9. Use a text editor, such as DOS's EDIT, to assemble the final
map from all the defragmented and smoothed .MAP files created for
each map feature type.  BOB COMMENTS:  I prefer to just use the
MAPFIX @IMPORT command to bring these into the working map...

  10. Run MAPFIX again and load the resulting complete .MAP file
with all your features. If it is too large, which it probably is,
delete points until you have a map which will fit within APRS
limits.

  - There is some small errors (less than 1/4 mile) toward the north east
and north west sections of the map. This is due to the conversion scheme
I use to get from the UTM grid to lat/long. The two grids do not line up
exactly. When you join two adjacent 15' maps, you will see that the roads
are off by about 100 meters. Use mapfix to tweak the lines so they join.


Please send me any program enhancements/fixes you make to either
program. I'm sure there are a lot better ways to defragment the
.MAP files - there MUST be!

PS - watch out for "Streams" - when I extract the ones around here
the resulting .MAP has thousands of points!

