diff --git a/.github/workflows/core_tests.yml b/.github/workflows/core_tests.yml index ef4b4acaf4..a1a8bbb1db 100644 --- a/.github/workflows/core_tests.yml +++ b/.github/workflows/core_tests.yml @@ -81,6 +81,7 @@ jobs: run: | python -m pytest --pyargs activitysim.cli + cross-platform: # also test foundation cross platforms, but do not require a successful # completion before starting regional model tests @@ -162,6 +163,7 @@ jobs: run: | python -m pytest --pyargs activitysim.cli + regional_models: needs: foundation env: diff --git a/.gitignore b/.gitignore index f89439651a..02e0f61d98 100644 --- a/.gitignore +++ b/.gitignore @@ -72,6 +72,7 @@ _test_est *_local/ *_local.* +**/__sharrowcache__ **/output/ **/_generated_version.py docs/**/_generated diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 4556a45a8f..3a59ed1529 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -3,8 +3,6 @@ repos: - repo: https://github.com/pre-commit/pre-commit-hooks rev: v4.3.0 hooks: - - id: check-yaml - exclude: logging.yaml # TODO don't exclude, will require fixing logging - id: end-of-file-fixer exclude: .*\.ipynb - id: trailing-whitespace diff --git a/HOW_TO_RELEASE.md b/HOW_TO_RELEASE.md index e8f2a42634..fdfc60719b 100644 --- a/HOW_TO_RELEASE.md +++ b/HOW_TO_RELEASE.md @@ -5,9 +5,9 @@ 01. Check that the branch you intend to release is passing tests on Travis. If it's not passing there you should not release it. -00. Start from a completely clean conda environment - and git repository. Assuming you have `conda` installed, you can do so - by starting where ActivitySim is not yet cloned (e.g. in an empty +00. Start from a completely clean conda environment + and git repository. Assuming you have `conda` installed, you can do so + by starting where ActivitySim is not yet cloned (e.g. in an empty directory) and running: ```sh conda create -n TEMP-ASIM-DEV python=3.9 git gh -c conda-forge --override-channels @@ -19,7 +19,7 @@ 00. Per project policy, code on the main branch should have been released, but if you are *preparing* a release then the code should be on the `develop` - branch. Switch to that branch now, and make sure it is synced to the + branch. Switch to that branch now, and make sure it is synced to the version on GitHub: ```sh git switch develop @@ -28,45 +28,45 @@ 00. Update your Conda environment for testing. We do not want to use an existing environment on your machine, as it may be out-of-date - and we want to make sure everything passes muster using the + and we want to make sure everything passes muster using the most up-to-date dependencies available. The following command - will update the active environment (we made this to be `TEMP-ASIM-DEV` + will update the active environment (we made this to be `TEMP-ASIM-DEV` if you followed the directions above). ```sh conda env update --file=conda-environments/activitysim-dev.yml ``` - If you add to the ActivitySim dependencies, make sure to also update - the environments in `conda-environments`, which are used for testing - and development. If they are not updated, these environments will end + If you add to the ActivitySim dependencies, make sure to also update + the environments in `conda-environments`, which are used for testing + and development. If they are not updated, these environments will end up with dependencies loaded from *pip* instead of *conda-forge*. -00. Run pycodestyle to ensure that the codebase passes all style checks. +00. Run black to ensure that the codebase passes all style checks. This check should only take a few seconds. These checks are also done on Travis and are platform independent, so they should not be necessary to replicate locally, but are listed here for completeness. ```sh - pycodestyle . + black --check --diff . ``` 00. Run the regular test suite on Windows. Travis tests are done on Linux, but most users are on Windows, and the test suite should also be run on Windows to ensure that it works on that platform as well. If you are not preparing this release on Windows, you should be sure to run - at least through this step on a Windows machine before finalizing a - release. - - A few of the tests require pre-created data that is not included in the - repository directly, but rather recreated on the fly before testing. The - regular test suite takes some time to run, between about half an hour and + at least through this step on a Windows machine before finalizing a + release. + + A few of the tests require pre-created data that is not included in the + repository directly, but rather recreated on the fly before testing. The + regular test suite takes some time to run, between about half an hour and two hours depending on the specs of your machine. ```sh python activitysim/examples/placeholder_multiple_zone/scripts/two_zone_example_data.py python activitysim/examples/placeholder_multiple_zone/scripts/three_zone_example_data.py pytest . ``` - + 00. Test the full-scale regional examples. These examples are big, too - large to run on Travis, and will take a lot of time (many hours) to + large to run on Travis, and will take a lot of time (many hours) to download and run. ```sh mkdir tmp-asim @@ -76,7 +76,7 @@ call run_all_examples.bat ``` These tests will run through the gamut even if some of them crash, so - if you don't sit and watch them go (please don't do this) you'll need + if you don't sit and watch them go (please don't do this) you'll need to scan through the results to make sure there are no errors after the fact. ```sh @@ -84,31 +84,31 @@ ``` 00. Test the notebooks in `activitysim/examples/prototype_mtc/notebooks`. - There are also demo notebooks for estimation, but their functionality + There are also demo notebooks for estimation, but their functionality is completely tested in the unit tests run previously. -00. Use bump2version to tag the release commit and update the +00. Use bump2version to tag the release commit and update the version number. The following code will generate a "patch" release, - incrementing the third value in the version number (i.e. "1.2.3" - becomes "1.2.4"). Alternatively, make a "minor" or "major" release. - The `--list` command will generate output to your console to confirm - that the old and new version numbers are what you expect, before you - push the commit (with the changed version in the code) and tags to + incrementing the third value in the version number (i.e. "1.2.3" + becomes "1.2.4"). Alternatively, make a "minor" or "major" release. + The `--list` command will generate output to your console to confirm + that the old and new version numbers are what you expect, before you + push the commit (with the changed version in the code) and tags to GitHub. ```sh bump2version patch --list ``` - - It is also possible to make a development pre-release. To do so, - explicitly set the version number to the next patch plus a ".devN" - suffix: - + + It is also possible to make a development pre-release. To do so, + explicitly set the version number to the next patch plus a ".devN" + suffix: + ```sh bump2version patch --new-version 1.2.3.dev0 --list ``` - - Then, when ready to make a "final" release, set the version by - explicitly removing the suffix: + + Then, when ready to make a "final" release, set the version by + explicitly removing the suffix: ```sh bump2version patch --new-version 1.2.3 --list ``` @@ -118,26 +118,26 @@ git push --tags ``` -00. For non-development releases, open a pull request to merge the proposed - release into main. The following command will open a web browser for +00. For non-development releases, open a pull request to merge the proposed + release into main. The following command will open a web browser for you to create the pull request. ```sh gh pr create --web ``` After creating the PR, confirm with the ActivitySim PMC that the release is ready before actually merging it. - + Once final approval is granted, merge the PR into main. The presence of the git tags added earlier will trigger automated build steps to prepare and deploy the release to pypi and conda-forge. - + 00. Create a "release" on GitHub. ```sh gh release create v1.2.3 ``` For a development pre-release, include the `--prerelease` argument. As the project's policy is that only formally released code is merged - to the main branch, any pre-release should also be built against a + to the main branch, any pre-release should also be built against a non-default branch. For example, to pre-release from the `develop` branch: ```sh @@ -147,11 +147,11 @@ --notes "this pre-release is for a cool new feature" \ --title "Development Pre-Release" ``` - -00. Clean up your workspace, including removing the Conda environment used for - testing (which will prevent you from accidentally using an old + +00. Clean up your workspace, including removing the Conda environment used for + testing (which will prevent you from accidentally using an old environment when you should have a fresh up-to-date one next time). ```sh conda deactivate conda env remove -n TEMP-ASIM-DEV - ``` \ No newline at end of file + ``` diff --git a/activitysim/abm/models/__init__.py b/activitysim/abm/models/__init__.py index 76e6855635..41ddbff18c 100644 --- a/activitysim/abm/models/__init__.py +++ b/activitysim/abm/models/__init__.py @@ -26,9 +26,12 @@ parking_location_choice, stop_frequency, summarize, + telecommute_frequency, tour_mode_choice, tour_od_choice, tour_scheduling_probabilistic, + transit_pass_ownership, + transit_pass_subsidy, trip_departure_choice, trip_destination, trip_matrices, @@ -39,4 +42,5 @@ trip_scheduling_choice, vehicle_allocation, vehicle_type_choice, + work_from_home, ) diff --git a/activitysim/abm/models/initialize_los.py b/activitysim/abm/models/initialize_los.py index 63e2f415f3..2649c09a3b 100644 --- a/activitysim/abm/models/initialize_los.py +++ b/activitysim/abm/models/initialize_los.py @@ -154,8 +154,16 @@ def compute_utilities_for_attribute_tuple( choosers_df, chunk_size, trace_label, chunk_tag=chunk_tag ): # we should count choosers_df as chunk overhead since its pretty big and was custom made for compute_utilities - assert chooser_chunk._is_view # otherwise copying it is wasteful - chooser_chunk = chooser_chunk.copy() + if chooser_chunk._is_view: + chooser_chunk = ( + chooser_chunk.copy() + ) # copy is needed when we start with a view + else: + # copying this is wasteful, but the code below edits the dataframe, + # which could have undesirable side effects. + # TODO: convert to Dataset or otherwise stop this copying, without + # harming anything else. + chooser_chunk = chooser_chunk.copy() chunk.log_df(trace_label, "attribute_chooser_chunk", chooser_chunk) # add any attribute columns specified as column attributes in settings (the rest will be scalars in locals_dict) diff --git a/activitysim/abm/models/joint_tour_participation.py b/activitysim/abm/models/joint_tour_participation.py index 7ef4d43af7..939d9feddb 100644 --- a/activitysim/abm/models/joint_tour_participation.py +++ b/activitysim/abm/models/joint_tour_participation.py @@ -133,6 +133,8 @@ def participants_chooser(probs, choosers, spec, trace_label): In principal, this shold always occur eventually, but we fail after MAX_ITERATIONS, just in case there is some failure in program logic (haven't seen this occur.) + The return values are the same as logit.make_choices + Parameters ---------- probs : pandas.DataFrame diff --git a/activitysim/abm/models/location_choice.py b/activitysim/abm/models/location_choice.py index f35d3eb158..3e806512ce 100644 --- a/activitysim/abm/models/location_choice.py +++ b/activitysim/abm/models/location_choice.py @@ -2,7 +2,6 @@ # See full license in LICENSE.txt. import logging -import numpy as np import pandas as pd from activitysim.abm.tables import shadow_pricing @@ -10,9 +9,7 @@ config, expressions, inject, - logit, los, - mem, pipeline, simulate, tracing, @@ -116,6 +113,7 @@ def _location_sample( chunk_size, chunk_tag, trace_label, + zone_layer=None, ): """ select a sample of alternative locations. @@ -149,7 +147,13 @@ def _location_sample( ) sample_size = 0 - locals_d = {"skims": skims, "segment_size": segment_name} + locals_d = { + "skims": skims, + "segment_size": segment_name, + "orig_col_name": skims.orig_key, # added for sharrow flows + "dest_col_name": skims.dest_key, # added for sharrow flows + "timeframe": "timeless", + } constants = config.get_model_constants(model_settings) locals_d.update(constants) @@ -175,6 +179,7 @@ def _location_sample( chunk_size=chunk_size, chunk_tag=chunk_tag, trace_label=trace_label, + zone_layer=zone_layer, ) return choices @@ -232,16 +237,17 @@ def aggregate_size_terms(dest_size_terms, network_los): # aggregate MAZ_size_terms to TAZ_size_terms # + # TODO:SHARROW FIX ME + sharrow_enabled = config.setting("sharrow", False) + if sharrow_enabled: + skim_dataset = inject.get_injectable("skim_dataset") + else: + skim_dataset = None + MAZ_size_terms = dest_size_terms.copy() # add crosswalk DEST_TAZ column to MAZ_size_terms - maz_to_taz = ( - network_los.maz_taz_df[["MAZ", "TAZ"]] - .set_index("MAZ") - .sort_values(by="TAZ") - .TAZ - ) - MAZ_size_terms[DEST_TAZ] = MAZ_size_terms.index.map(maz_to_taz) + MAZ_size_terms[DEST_TAZ] = network_los.map_maz_to_taz(MAZ_size_terms.index) weighted_average_cols = [ "shadow_price_size_term_adjustment", @@ -341,6 +347,7 @@ def location_presample( chunk_size, chunk_tag, trace_label, + zone_layer="taz", ) # print(f"taz_sample\n{taz_sample}") @@ -563,7 +570,13 @@ def run_location_simulate( skim_dict = network_los.get_default_skim_dict() skims = skim_dict.wrap("home_zone_id", alt_dest_col_name) - locals_d = {"skims": skims, "segment_size": segment_name} + locals_d = { + "skims": skims, + "segment_size": segment_name, + "orig_col_name": skims.orig_key, # added for sharrow flows + "dest_col_name": skims.dest_key, # added for sharrow flows + "timeframe": "timeless", + } constants = config.get_model_constants(model_settings) if constants is not None: locals_d.update(constants) diff --git a/activitysim/abm/models/parking_location_choice.py b/activitysim/abm/models/parking_location_choice.py index bddbc65190..f38688fdf2 100644 --- a/activitysim/abm/models/parking_location_choice.py +++ b/activitysim/abm/models/parking_location_choice.py @@ -122,6 +122,8 @@ def parking_destination_simulate( locals_dict = config.get_model_constants(model_settings).copy() locals_dict.update(skims) + locals_dict["timeframe"] = "trip" + locals_dict["PARKING"] = skims["op_skims"].dest_key parking_locations = interaction_sample_simulate( choosers=trips, @@ -296,6 +298,21 @@ def parking_location( trips_merged_df = trips_merged.to_frame() land_use_df = land_use.to_frame() + proposed_trip_departure_period = model_settings["TRIP_DEPARTURE_PERIOD"] + # TODO: the number of skim time periods should be more readily available than this + n_skim_time_periods = np.unique( + network_los.los_settings["skim_time_periods"]["labels"] + ).size + if trips_merged_df[proposed_trip_departure_period].max() > n_skim_time_periods: + # max proposed_trip_departure_period is out of range, + # it is most likely the high-resolution time period, we need the skim-level time period + if "trip_period" not in trips_merged_df: + # TODO: resolve this to the skim time period index not the label, it will be faster + trips_merged_df["trip_period"] = network_los.skim_time_period_label( + trips_merged_df[proposed_trip_departure_period] + ) + model_settings["TRIP_DEPARTURE_PERIOD"] = "trip_period" + locals_dict = {"network_los": network_los} locals_dict.update(config.get_model_constants(model_settings)) diff --git a/activitysim/examples/prototype_semcog/extensions/telecommute_frequency.py b/activitysim/abm/models/telecommute_frequency.py similarity index 100% rename from activitysim/examples/prototype_semcog/extensions/telecommute_frequency.py rename to activitysim/abm/models/telecommute_frequency.py diff --git a/activitysim/examples/prototype_semcog/extensions/transit_pass_ownership.py b/activitysim/abm/models/transit_pass_ownership.py similarity index 100% rename from activitysim/examples/prototype_semcog/extensions/transit_pass_ownership.py rename to activitysim/abm/models/transit_pass_ownership.py diff --git a/activitysim/examples/prototype_semcog/extensions/transit_pass_subsidy.py b/activitysim/abm/models/transit_pass_subsidy.py similarity index 100% rename from activitysim/examples/prototype_semcog/extensions/transit_pass_subsidy.py rename to activitysim/abm/models/transit_pass_subsidy.py diff --git a/activitysim/abm/models/trip_destination.py b/activitysim/abm/models/trip_destination.py index 133e5c5b06..d6bb98d600 100644 --- a/activitysim/abm/models/trip_destination.py +++ b/activitysim/abm/models/trip_destination.py @@ -12,7 +12,6 @@ ) from activitysim.abm.tables.size_terms import tour_destination_size_terms from activitysim.core import ( - assign, chunk, config, expressions, @@ -24,7 +23,6 @@ ) from activitysim.core.interaction_sample import interaction_sample from activitysim.core.interaction_sample_simulate import interaction_sample_simulate -from activitysim.core.pathbuilder import TransitVirtualPathBuilder from activitysim.core.skim_dictionary import DataFrameMatrix from activitysim.core.tracing import print_elapsed_time from activitysim.core.util import assign_in_place, reindex @@ -53,6 +51,7 @@ def _destination_sample( chunk_size, chunk_tag, trace_label, + zone_layer=None, ): """ @@ -95,7 +94,13 @@ def _destination_sample( # cannot be determined until after choosers are joined with alternatives # (unless we iterate over trip.purpose - which we could, though we are already iterating over trip_num) # so, instead, expressions determine row-specific size_term by a call to: size_terms.get(df.alt_dest, df.purpose) - locals_dict.update({"size_terms": size_term_matrix}) + locals_dict.update( + { + "size_terms": size_term_matrix, + "size_terms_array": size_term_matrix.df.to_numpy(), + "timeframe": "trip", + } + ) locals_dict.update(skims) log_alt_losers = config.setting("log_alt_losers", False) @@ -113,6 +118,7 @@ def _destination_sample( chunk_size=chunk_size, chunk_tag=chunk_tag, trace_label=trace_label, + zone_layer=zone_layer, ) return choices @@ -152,12 +158,12 @@ def destination_sample( return choices -def aggregate_size_term_matrix(maz_size_term_matrix, maz_taz): +def aggregate_size_term_matrix(maz_size_term_matrix, network_los): df = maz_size_term_matrix.df assert ALT_DEST_TAZ not in df - dest_taz = df.index.map(maz_taz) + dest_taz = network_los.map_maz_to_taz(df.index) taz_size_term_matrix = df.groupby(dest_taz).sum() taz_size_term_matrix = DataFrameMatrix(taz_size_term_matrix) @@ -256,9 +262,13 @@ def choose_MAZ_for_TAZ( # there will be a different set (and number) of candidate MAZs for each TAZ # (preserve index, which will have duplicates as result of join) - maz_taz = network_los.maz_taz_df[["MAZ", "TAZ"]].rename( - columns={"TAZ": DEST_TAZ, "MAZ": DEST_MAZ} + maz_taz = ( + network_los.get_maz_to_taz_series.rename(DEST_TAZ) + .rename_axis(index=DEST_MAZ) + .to_frame() + .reset_index() ) + maz_sizes = pd.merge( taz_choices[[chooser_id_col, DEST_TAZ]].reset_index(), maz_taz, @@ -426,20 +436,21 @@ def destination_presample( chunk_tag = "trip_destination.presample" # distinguish from trip_destination.sample alt_dest_col_name = model_settings["ALT_DEST_COL_NAME"] - maz_taz = network_los.maz_taz_df[["MAZ", "TAZ"]].set_index("MAZ").TAZ - TAZ_size_term_matrix = aggregate_size_term_matrix(size_term_matrix, maz_taz) + TAZ_size_term_matrix = aggregate_size_term_matrix(size_term_matrix, network_los) TRIP_ORIGIN = model_settings["TRIP_ORIGIN"] PRIMARY_DEST = model_settings["PRIMARY_DEST"] trips_taz = trips.copy() - trips_taz[TRIP_ORIGIN] = trips_taz[TRIP_ORIGIN].map(maz_taz) - trips_taz[PRIMARY_DEST] = trips_taz[PRIMARY_DEST].map(maz_taz) + trips_taz[TRIP_ORIGIN] = network_los.map_maz_to_taz(trips_taz[TRIP_ORIGIN]) + trips_taz[PRIMARY_DEST] = network_los.map_maz_to_taz(trips_taz[PRIMARY_DEST]) # alternatives is just an empty dataframe indexed by maz with index name # but logically, we are aggregating so lets do it, as there is no particular gain in being clever - alternatives = alternatives.groupby(alternatives.index.map(maz_taz)).sum() + alternatives = alternatives.groupby( + network_los.map_maz_to_taz(alternatives.index) + ).sum() # # i did this but after changing alt_dest_col_name to 'trip_dest' it # # shouldn't be needed anymore @@ -459,6 +470,7 @@ def destination_presample( chunk_size, chunk_tag=chunk_tag, trace_label=trace_label, + zone_layer="taz", ) # choose a MAZ for each DEST_TAZ choice, choice probability based on MAZ size_term fraction of TAZ total @@ -677,6 +689,7 @@ def compute_logsums( "odt_skims": skims["odt_skims"], "dot_skims": skims["dot_skims"], "od_skims": skims["od_skims"], + "timeframe": "trip", } if network_los.zone_system == los.THREE_ZONE: od_skims.update( @@ -769,8 +782,20 @@ def trip_destination_simulate( skims = skim_hotel.sample_skims(presample=False) + if not np.issubdtype(trips["trip_period"].dtype, np.integer): + if hasattr(skims["odt_skims"], "map_time_periods"): + trip_period_idx = skims["odt_skims"].map_time_periods(trips) + if trip_period_idx is not None: + trips["trip_period"] = trip_period_idx + locals_dict = config.get_model_constants(model_settings).copy() - locals_dict.update({"size_terms": size_term_matrix}) + locals_dict.update( + { + "size_terms": size_term_matrix, + "size_terms_array": size_term_matrix.df.to_numpy(), + "timeframe": "trip", + } + ) locals_dict.update(skims) log_alt_losers = config.setting("log_alt_losers", False) @@ -1168,7 +1193,10 @@ def run_trip_destination( trace_label, "trip_num_%s" % trip_num ) - locals_dict = {"network_los": network_los} + locals_dict = { + "network_los": network_los, + "size_terms": size_term_matrix, + } locals_dict.update(config.get_model_constants(model_settings)) # - annotate nth_trips @@ -1180,6 +1208,15 @@ def run_trip_destination( trace_label=nth_trace_label, ) + if not np.issubdtype(nth_trips["trip_period"].dtype, np.integer): + skims = network_los.get_default_skim_dict() + if hasattr(skims, "map_time_periods_from_series"): + trip_period_idx = skims.map_time_periods_from_series( + nth_trips["trip_period"] + ) + if trip_period_idx is not None: + nth_trips["trip_period"] = trip_period_idx + logger.info("Running %s with %d trips", nth_trace_label, nth_trips.shape[0]) # - choose destination for nth_trips, segmented by primary_purpose @@ -1318,7 +1355,7 @@ def trip_destination(trips, tours_merged, chunk_size, trace_hh_id): ): if (trips_df.trip_num < trips_df.trip_count).sum() == 0: raise RuntimeError( - f"can't honor 'testing_fail_trip_destination' setting because no intermediate trips" + "can't honor 'testing_fail_trip_destination' setting because no intermediate trips" ) fail_o = trips_df[trips_df.trip_num < trips_df.trip_count].origin.max() diff --git a/activitysim/abm/models/trip_matrices.py b/activitysim/abm/models/trip_matrices.py index 0f33e69e69..d076ac8ac3 100644 --- a/activitysim/abm/models/trip_matrices.py +++ b/activitysim/abm/models/trip_matrices.py @@ -38,7 +38,7 @@ def write_trip_matrices(network_los): # this might legitimately happen if they comment out some steps to debug but still want write_tables # this saves them the hassle of remembering to comment out this step logger.warning( - f"write_trip_matrices returning empty-handed because there is no trips table" + "write_trip_matrices returning empty-handed because there is no trips table" ) return @@ -52,6 +52,7 @@ def write_trip_matrices(network_los): parking_settings = config.read_model_settings("parking_location_choice.yaml") parking_taz_col_name = parking_settings["ALT_DEST_COL_NAME"] if parking_taz_col_name in trips_df: + # TODO make parking zone negative, not zero, if not used trips_df.loc[trips_df[parking_taz_col_name] > 0, "destination"] = trips_df[ parking_taz_col_name ] @@ -75,15 +76,21 @@ def write_trip_matrices(network_los): dest_vals = aggregate_trips.index.get_level_values("destination") # use the land use table for the set of possible tazs - zone_index = pipeline.get_table("land_use").index + land_use = pipeline.get_table("land_use") + zone_index = land_use.index assert all(zone in zone_index for zone in orig_vals) assert all(zone in zone_index for zone in dest_vals) _, orig_index = zone_index.reindex(orig_vals) _, dest_index = zone_index.reindex(dest_vals) + try: + zone_labels = land_use[f"_original_{land_use.index.name}"] + except KeyError: + zone_labels = land_use.index + write_matrices( - aggregate_trips, zone_index, orig_index, dest_index, model_settings + aggregate_trips, zone_labels, orig_index, dest_index, model_settings ) elif network_los.zone_system == los.TWO_ZONE: # maz trips written to taz matrices @@ -108,6 +115,15 @@ def write_trip_matrices(network_los): orig_vals = aggregate_trips.index.get_level_values("otaz") dest_vals = aggregate_trips.index.get_level_values("dtaz") + try: + land_use_taz = pipeline.get_table("land_use_taz") + except (KeyError, RuntimeError): + pass # table missing, ignore + else: + if "_original_TAZ" in land_use_taz.columns: + orig_vals = orig_vals.map(land_use_taz["_original_TAZ"]) + dest_vals = dest_vals.map(land_use_taz["_original_TAZ"]) + zone_index = pd.Index(network_los.get_tazs(), name="TAZ") assert all(zone in zone_index for zone in orig_vals) assert all(zone in zone_index for zone in dest_vals) @@ -144,6 +160,15 @@ def write_trip_matrices(network_los): orig_vals = aggregate_trips.index.get_level_values("otaz") dest_vals = aggregate_trips.index.get_level_values("dtaz") + try: + land_use_taz = pipeline.get_table("land_use_taz") + except (KeyError, RuntimeError): + pass # table missing, ignore + else: + if "_original_TAZ" in land_use_taz.columns: + orig_vals = orig_vals.map(land_use_taz["_original_TAZ"]) + dest_vals = dest_vals.map(land_use_taz["_original_TAZ"]) + zone_index = pd.Index(network_los.get_tazs(), name="TAZ") assert all(zone in zone_index for zone in orig_vals) assert all(zone in zone_index for zone in dest_vals) @@ -216,6 +241,14 @@ def annotate_trips(trips, network_los, model_settings): trips_df, locals_dict, skims, model_settings, trace_label ) + if not np.issubdtype(trips_df["trip_period"].dtype, np.integer): + if hasattr(skim_dict, "map_time_periods_from_series"): + trip_period_idx = skim_dict.map_time_periods_from_series( + trips_df["trip_period"] + ) + if trip_period_idx is not None: + trips_df["trip_period"] = trip_period_idx + # Data will be expanded by an expansion weight column from # the households pipeline table, if specified in the model settings. hh_weight_col = model_settings.get("HH_EXPANSION_WEIGHT_COL") diff --git a/activitysim/abm/models/trip_mode_choice.py b/activitysim/abm/models/trip_mode_choice.py index 59bc8433f1..0fb2bc3818 100644 --- a/activitysim/abm/models/trip_mode_choice.py +++ b/activitysim/abm/models/trip_mode_choice.py @@ -102,6 +102,13 @@ def trip_mode_choice(trips, network_los, chunk_size, trace_hh_id): ) od_skim_wrapper = skim_dict.wrap("origin", "destination") + if hasattr(skim_dict, "map_time_periods_from_series"): + trip_period_idx = skim_dict.map_time_periods_from_series( + trips_merged["trip_period"] + ) + if trip_period_idx is not None: + trips_merged["trip_period"] = trip_period_idx + skims = { "odt_skims": odt_skim_stack_wrapper, "dot_skims": dot_skim_stack_wrapper, @@ -204,6 +211,7 @@ def trip_mode_choice(trips, network_los, chunk_size, trace_hh_id): estimator.write_choosers(trips_segment) locals_dict.update(skims) + locals_dict["timeframe"] = "trip" choices = mode_choice_simulate( choosers=trips_segment, diff --git a/activitysim/abm/models/trip_scheduling_choice.py b/activitysim/abm/models/trip_scheduling_choice.py index 463a00ee76..5a7abe1f53 100644 --- a/activitysim/abm/models/trip_scheduling_choice.py +++ b/activitysim/abm/models/trip_scheduling_choice.py @@ -370,6 +370,9 @@ def trip_scheduling_choice(trips, tours, skim_dict, chunk_size, trace_hh_id): "od_skims": od_skim_stack_wrapper, "do_skims": do_skim_stack_wrapper, "obib_skims": obib_skim_stack_wrapper, + "orig_col_name": "origin", + "dest_col_name": "destination", + "timeframe": "timeless_directional", } simulate.set_skim_wrapper_targets(tours_df, skims) diff --git a/activitysim/abm/models/util/tour_destination.py b/activitysim/abm/models/util/tour_destination.py index a39037b479..cc8f660fed 100644 --- a/activitysim/abm/models/util/tour_destination.py +++ b/activitysim/abm/models/util/tour_destination.py @@ -76,6 +76,7 @@ def _destination_sample( chunk_size, chunk_tag, trace_label, + zone_layer=None, ): model_spec = simulate.spec_for_segment( @@ -98,7 +99,12 @@ def _destination_sample( ) sample_size = 0 - locals_d = {"skims": skims} + locals_d = { + "skims": skims, + "orig_col_name": skims.orig_key, # added for sharrow flows + "dest_col_name": skims.dest_key, # added for sharrow flows + "timeframe": "timeless", + } constants = config.get_model_constants(model_settings) if constants is not None: locals_d.update(constants) @@ -117,6 +123,7 @@ def _destination_sample( chunk_size=chunk_size, chunk_tag=chunk_tag, trace_label=trace_label, + zone_layer=zone_layer, ) # remember person_id in chosen alts so we can merge with persons in subsequent steps @@ -175,11 +182,6 @@ def destination_sample( ORIG_TAZ = "TAZ" # likewise a temp, but if already in choosers, we assume we can use it opportunistically -def map_maz_to_taz(s, network_los): - maz_to_taz = network_los.maz_taz_df[["MAZ", "TAZ"]].set_index("MAZ").TAZ - return s.map(maz_to_taz) - - def aggregate_size_terms(dest_size_terms, network_los): # # aggregate MAZ_size_terms to TAZ_size_terms @@ -188,7 +190,9 @@ def aggregate_size_terms(dest_size_terms, network_los): MAZ_size_terms = dest_size_terms.copy() # add crosswalk DEST_TAZ column to MAZ_size_terms - MAZ_size_terms[DEST_TAZ] = map_maz_to_taz(MAZ_size_terms.index, network_los) + MAZ_size_terms[DEST_TAZ] = network_los.map_maz_to_taz(MAZ_size_terms.index) + if MAZ_size_terms[DEST_TAZ].isna().any(): + raise ValueError("found NaN MAZ") # aggregate to TAZ TAZ_size_terms = MAZ_size_terms.groupby(DEST_TAZ).agg({"size_term": "sum"}) @@ -216,6 +220,9 @@ def aggregate_size_terms(dest_size_terms, network_los): # print(f"TAZ_size_terms ({TAZ_size_terms.shape})\n{TAZ_size_terms}") # print(f"MAZ_size_terms ({MAZ_size_terms.shape})\n{MAZ_size_terms}") + if np.issubdtype(TAZ_size_terms[DEST_TAZ], np.floating): + raise TypeError("TAZ indexes are not integer") + return MAZ_size_terms, TAZ_size_terms @@ -475,7 +482,7 @@ def destination_presample( orig_maz = model_settings["CHOOSER_ORIG_COL_NAME"] assert orig_maz in choosers if ORIG_TAZ not in choosers: - choosers[ORIG_TAZ] = map_maz_to_taz(choosers[orig_maz], network_los) + choosers[ORIG_TAZ] = network_los.map_maz_to_taz(choosers[orig_maz]) # create wrapper with keys for this lookup - in this case there is a HOME_TAZ in the choosers # and a DEST_TAZ in the alternatives which get merged during interaction @@ -494,6 +501,7 @@ def destination_presample( chunk_size, chunk_tag=chunk_tag, trace_label=trace_label, + zone_layer="taz", ) # choose a MAZ for each DEST_TAZ choice, choice probability based on MAZ size_term fraction of TAZ total @@ -721,6 +729,9 @@ def run_destination_simulate( locals_d = { "skims": skims, + "orig_col_name": skims.orig_key, # added for sharrow flows + "dest_col_name": skims.dest_key, # added for sharrow flows + "timeframe": "timeless", } if constants is not None: locals_d.update(constants) diff --git a/activitysim/abm/models/util/tour_od.py b/activitysim/abm/models/util/tour_od.py index 55cc9a4a00..91377bd6db 100644 --- a/activitysim/abm/models/util/tour_od.py +++ b/activitysim/abm/models/util/tour_od.py @@ -156,7 +156,12 @@ def _od_sample( ) sample_size = 0 - locals_d = {"skims": skims} + locals_d = { + "skims": skims, + "timeframe": "timeless", + "orig_col_name": ORIG_TAZ, + "dest_col_name": DEST_TAZ, + } constants = config.get_model_constants(model_settings) if constants is not None: locals_d.update(constants) @@ -175,7 +180,7 @@ def _od_sample( ) if skims.orig_key == ORIG_TAZ: - od_alts_df[ORIG_TAZ] = map_maz_to_taz(od_alts_df[origin_id_col], network_los) + od_alts_df[ORIG_TAZ] = network_los.map_maz_to_taz(od_alts_df[origin_id_col]) elif skims.orig_key not in od_alts_df: logger.error("Alts df is missing origin skim key column.") @@ -191,6 +196,7 @@ def _od_sample( chunk_size=chunk_size, chunk_tag=chunk_tag, trace_label=trace_label, + zone_layer="taz", ) return choices @@ -274,7 +280,7 @@ def aggregate_size_terms(dest_size_terms, network_los): MAZ_size_terms = dest_size_terms.copy() # add crosswalk DEST_TAZ column to MAZ_size_terms - MAZ_size_terms[DEST_TAZ] = map_maz_to_taz(MAZ_size_terms.index, network_los) + MAZ_size_terms[DEST_TAZ] = network_los.map_maz_to_taz(MAZ_size_terms.index) # aggregate to TAZ TAZ_size_terms = MAZ_size_terms.groupby(DEST_TAZ).agg({"size_term": "sum"}) @@ -993,6 +999,9 @@ def run_od_simulate( locals_d = { "skims": skims, + "timeframe": "timeless", + "orig_col_name": origin_col_name, + "dest_col_name": dest_col_name, } if constants is not None: locals_d.update(constants) diff --git a/activitysim/abm/models/util/vectorize_tour_scheduling.py b/activitysim/abm/models/util/vectorize_tour_scheduling.py index b7698ae11d..83cb6c547a 100644 --- a/activitysim/abm/models/util/vectorize_tour_scheduling.py +++ b/activitysim/abm/models/util/vectorize_tour_scheduling.py @@ -5,20 +5,10 @@ import numpy as np import pandas as pd -from activitysim.core import ( - chunk, - config, - expressions, - inject, - logit, - los, - mem, - simulate, -) +from activitysim.core import chunk, config, expressions, inject, los, simulate from activitysim.core import timetable as tt from activitysim.core import tracing from activitysim.core.interaction_sample_simulate import interaction_sample_simulate -from activitysim.core.pathbuilder import TransitVirtualPathBuilder from activitysim.core.util import reindex logger = logging.getLogger(__name__) @@ -179,7 +169,7 @@ def dedupe_alt_tdd(alt_tdd, tour_purpose, trace_label): tdd_segments = inject.get_injectable("tdd_alt_segments", None) alt_tdd_periods = None - logger.info(f"tdd_alt_segments specified for representative logsums") + logger.info("tdd_alt_segments specified for representative logsums") with chunk.chunk_log(tracing.extend_trace_label(trace_label, "dedupe_alt_tdd")): @@ -272,8 +262,8 @@ def dedupe_alt_tdd(alt_tdd, tour_purpose, trace_label): dedupe_columns = ["out_period", "in_period", "duration"] logger.warning( - f"No tdd_alt_segments for representative logsums so fallback to " - f"deduping tdd_alts by time_period and duration" + "No tdd_alt_segments for representative logsums so fallback to " + "deduping tdd_alts by time_period and duration" ) # - get list of unique (tour_id, out_period, in_period, duration) in alt_tdd_periods @@ -317,8 +307,13 @@ def compute_logsums( # - in_period and out_period assert "out_period" not in alt_tdd assert "in_period" not in alt_tdd + + # FIXME:MEMORY + # These two lines each generate a massive array of strings, + # using a bunch of RAM and slowing things down. alt_tdd["out_period"] = network_los.skim_time_period_label(alt_tdd["start"]) alt_tdd["in_period"] = network_los.skim_time_period_label(alt_tdd["end"]) + alt_tdd["duration"] = alt_tdd["end"] - alt_tdd["start"] # outside chunk_log context because we extend log_df call for alt_tdd made by our only caller _schedule_tours @@ -352,7 +347,7 @@ def compute_logsums( f"from {len(alt_tdd)} to {len(deduped_alt_tdds)} compared to USE_BRUTE_FORCE_TO_COMPUTE_LOGSUMS" ) - t0 = tracing.print_elapsed_time() + tracing.print_elapsed_time() # - compute logsums for the alt_tdd_periods deduped_alt_tdds["logsums"] = _compute_logsums( @@ -458,9 +453,9 @@ def tdd_interaction_dataset( Parameters ---------- - tours : pandas DataFrame + tours : pandas.DataFrame must have person_id column and index on tour_id - alts : pandas DataFrame + alts : pandas.DataFrame alts index must be timetable tdd id timetable : TimeTable object choice_column : str @@ -484,35 +479,46 @@ def tdd_interaction_dataset( tour_ids = np.repeat(tours.index, len(alts.index)) window_row_ids = np.repeat(tours[window_id_col], len(alts.index)) + chunk.log_df(trace_label, "window_row_ids", window_row_ids) alt_tdd = alts.take(alts_ids) alt_tdd.index = tour_ids - alt_tdd[window_id_col] = window_row_ids + + import xarray as xr + + alt_tdd_ = xr.Dataset.from_dataframe(alt_tdd) + dimname = alt_tdd.index.name or "index" + # alt_tdd_[window_id_col] = xr.DataArray(window_row_ids, dims=(dimname,)) + alt_tdd_[choice_column] = xr.DataArray( + alts_ids, dims=(dimname,), coords=alt_tdd_.coords + ) # add tdd alternative id # by convention, the choice column is the first column in the interaction dataset - alt_tdd.insert(loc=0, column=choice_column, value=alts_ids) + # alt_tdd.insert(loc=0, column=choice_column, value=alts_ids) # slice out all non-available tours - available = timetable.tour_available( - alt_tdd[window_id_col], alt_tdd[choice_column] - ) + available = timetable.tour_available(window_row_ids, alts_ids) + + del window_row_ids + chunk.log_df(trace_label, "window_row_ids", None) + logger.debug( f"tdd_interaction_dataset keeping {available.sum()} of ({len(available)}) available alt_tdds" ) assert available.any() chunk.log_df( - trace_label, "alt_tdd", alt_tdd + trace_label, "alt_tdd_", alt_tdd_ ) # catch this before we slice on available - alt_tdd = alt_tdd[available] + alt_tdd = alt_tdd_.isel({dimname: available}).to_dataframe() chunk.log_df(trace_label, "alt_tdd", alt_tdd) # FIXME - don't need this any more after slicing - del alt_tdd[window_id_col] + # del alt_tdd[window_id_col] return alt_tdd @@ -814,7 +820,7 @@ def schedule_tours( result_list.append(choices) - chunk.log_df(tour_trace_label, f"result_list", result_list) + chunk.log_df(tour_trace_label, "result_list", result_list) # FIXME: this will require 2X RAM # if necessary, could append to hdf5 store on disk: diff --git a/activitysim/examples/prototype_semcog/extensions/work_from_home.py b/activitysim/abm/models/work_from_home.py similarity index 100% rename from activitysim/examples/prototype_semcog/extensions/work_from_home.py rename to activitysim/abm/models/work_from_home.py diff --git a/activitysim/abm/tables/accessibility.py b/activitysim/abm/tables/accessibility.py index 3fd0544216..6869da736c 100644 --- a/activitysim/abm/tables/accessibility.py +++ b/activitysim/abm/tables/accessibility.py @@ -31,9 +31,23 @@ def accessibility(land_use): "created placeholder accessibility table %s" % (accessibility_df.shape,) ) else: - assert accessibility_df.sort_index().index.equals( - land_use.to_frame().sort_index().index - ), f"loaded accessibility table index does not match index of land_use table" + try: + assert accessibility_df.sort_index().index.equals( + land_use.to_frame().sort_index().index + ), f"loaded accessibility table index does not match index of land_use table" + except AssertionError: + land_use_index = land_use.to_frame().index + if f"_original_{land_use_index.name}" in land_use.to_frame(): + land_use_zone_ids = land_use.to_frame()[ + f"_original_{land_use_index.name}" + ] + remapper = dict(zip(land_use_zone_ids, land_use_zone_ids.index)) + accessibility_df.index = accessibility_df.index.map(remapper.get) + assert accessibility_df.sort_index().index.equals( + land_use.to_frame().sort_index().index + ), f"loaded accessibility table index does not match index of land_use table" + else: + raise logger.info("loaded land_use %s" % (accessibility_df.shape,)) # replace table function with dataframe diff --git a/activitysim/abm/tables/households.py b/activitysim/abm/tables/households.py index 9c115f17ce..e0a42f63bc 100644 --- a/activitysim/abm/tables/households.py +++ b/activitysim/abm/tables/households.py @@ -1,5 +1,6 @@ # ActivitySim # See full license in LICENSE.txt. +import io import logging from builtins import range @@ -96,6 +97,9 @@ def households(households_sample_size, override_hh_ids, trace_hh_id): df["sample_rate"] = sample_rate logger.info("loaded households %s" % (df.shape,)) + buffer = io.StringIO() + df.info(buf=buffer) + logger.debug("households.info:\n" + buffer.getvalue()) # replace table function with dataframe inject.add_table("households", df) diff --git a/activitysim/abm/tables/landuse.py b/activitysim/abm/tables/landuse.py index cf4e72420e..b74bc37422 100644 --- a/activitysim/abm/tables/landuse.py +++ b/activitysim/abm/tables/landuse.py @@ -1,5 +1,6 @@ # ActivitySim # See full license in LICENSE.txt. +import io import logging import pandas as pd @@ -22,6 +23,9 @@ def land_use(): df = df.sort_index() logger.info("loaded land_use %s" % (df.shape,)) + buffer = io.StringIO() + df.info(buf=buffer) + logger.debug("land_use.info:\n" + buffer.getvalue()) # replace table function with dataframe inject.add_table("land_use", df) @@ -30,3 +34,22 @@ def land_use(): inject.broadcast("land_use", "households", cast_index=True, onto_on="home_zone_id") + + +@inject.table() +def land_use_taz(): + + df = read_input_table("land_use_taz") + + if not df.index.is_monotonic_increasing: + df = df.sort_index() + + logger.info("loaded land_use_taz %s" % (df.shape,)) + buffer = io.StringIO() + df.info(buf=buffer) + logger.debug("land_use_taz.info:\n" + buffer.getvalue()) + + # replace table function with dataframe + inject.add_table("land_use_taz", df) + + return df diff --git a/activitysim/abm/tables/persons.py b/activitysim/abm/tables/persons.py index a193b8f6dd..9946db5f0e 100644 --- a/activitysim/abm/tables/persons.py +++ b/activitysim/abm/tables/persons.py @@ -1,5 +1,6 @@ # ActivitySim # See full license in LICENSE.txt. +import io import logging import pandas as pd @@ -27,6 +28,9 @@ def persons(households, trace_hh_id): df = read_raw_persons(households) logger.info("loaded persons %s" % (df.shape,)) + buffer = io.StringIO() + df.info(buf=buffer) + logger.debug("persons.info:\n" + buffer.getvalue()) # replace table function with dataframe inject.add_table("persons", df) diff --git a/activitysim/abm/tables/skims.py b/activitysim/abm/tables/skims.py index f7a841d9ad..39440b29f2 100644 --- a/activitysim/abm/tables/skims.py +++ b/activitysim/abm/tables/skims.py @@ -50,4 +50,6 @@ def log_settings(): "num_processes", "resume_after", "trace_hh_id", + "memory_profile", + "instrument", ] diff --git a/activitysim/abm/test/test_misc/test_load_cached_accessibility.py b/activitysim/abm/test/test_misc/test_load_cached_accessibility.py index 505fef890a..721b441d4e 100644 --- a/activitysim/abm/test/test_misc/test_load_cached_accessibility.py +++ b/activitysim/abm/test/test_misc/test_load_cached_accessibility.py @@ -81,15 +81,17 @@ def test_load_cached_accessibility(): "initialize_households", ] - pipeline.run(models=_MODELS, resume_after=None) + try: + pipeline.run(models=_MODELS, resume_after=None) - accessibility_df = pipeline.get_table("accessibility") + accessibility_df = pipeline.get_table("accessibility") - assert "auPkRetail" in accessibility_df + assert "auPkRetail" in accessibility_df - pipeline.close_pipeline() - inject.clear_cache() - close_handlers() + finally: + pipeline.close_pipeline() + inject.clear_cache() + close_handlers() if __name__ == "__main__": diff --git a/activitysim/benchmarking/asv.conf.json b/activitysim/benchmarking/asv.conf.json index 1b1148955f..bb8ab65c30 100644 --- a/activitysim/benchmarking/asv.conf.json +++ b/activitysim/benchmarking/asv.conf.json @@ -85,7 +85,9 @@ "numba": [], "coverage": [], "pytest": [], - "cytoolz": [] + "cytoolz": [], + "zarr": [], + "sharrow": [] }, // Combinations of libraries/python versions can be excluded/included diff --git a/activitysim/benchmarking/benchmarks/mtc1full.py b/activitysim/benchmarking/benchmarks/mtc1full.py index 39115a9922..e35e330fce 100644 --- a/activitysim/benchmarking/benchmarks/mtc1full.py +++ b/activitysim/benchmarking/benchmarks/mtc1full.py @@ -52,6 +52,14 @@ VERSION = "1" +try: + from activitysim import __data_compatability__ +except ImportError: + __data_compatability__ = None +else: + OUTPUT_DIR = f"{OUTPUT_DIR}-{__data_compatability__}" + + def setup_cache(): template_setup_cache( EXAMPLE_NAME, diff --git a/activitysim/benchmarking/benchmarks/sandag_example.py b/activitysim/benchmarking/benchmarks/sandag_example.py index b9c74c86eb..0ba16a9ba2 100644 --- a/activitysim/benchmarking/benchmarks/sandag_example.py +++ b/activitysim/benchmarking/benchmarks/sandag_example.py @@ -33,6 +33,7 @@ ] BENCHMARK_SETTINGS = { "households_sample_size": 48_769, + "sharrow": True, } SKIM_CACHE = True PRELOAD_INJECTABLES = ("skim_dict",) diff --git a/activitysim/benchmarking/componentwise.py b/activitysim/benchmarking/componentwise.py index dda43c4548..bb3c909b08 100644 --- a/activitysim/benchmarking/componentwise.py +++ b/activitysim/benchmarking/componentwise.py @@ -63,6 +63,7 @@ def setup_component( data_dir="data", output_dir="output", settings_filename="settings.yaml", + **other_settings, ): """ Prepare to benchmark a model component. @@ -84,6 +85,7 @@ def setup_component( settings_filename, benchmarking=component_name, checkpoints=False, + **other_settings, ) component_logging(component_name) @@ -144,6 +146,7 @@ def setup_component( "use_shadow_pricing", "want_dest_choice_sample_tables", "log_alt_losers", + "sharrow", ] for k in log_settings: logger.info(f"SETTING {k}: {config.setting(k)}") @@ -623,6 +626,7 @@ def setup(self): config_dirs, data_dir, output_dir, + sharrow=True, ) def teardown(self): diff --git a/activitysim/benchmarking/instrument.py b/activitysim/benchmarking/instrument.py new file mode 100644 index 0000000000..ede07b2eff --- /dev/null +++ b/activitysim/benchmarking/instrument.py @@ -0,0 +1,28 @@ +import importlib +import os +import webbrowser + +from pyinstrument import Profiler + + +def run_instrument(bench_name, component_name, out_file=None): + bench_module = importlib.import_module( + f"activitysim.benchmarking.benchmarks.{bench_name}" + ) + + component = getattr(bench_module, component_name)() + component.setup() + with Profiler() as profiler: + component.time_component() + component.teardown() + + if out_file is None: + out_file = f"instrument/{bench_name}/{component_name}.html" + dirname = os.path.dirname(out_file) + if dirname: + os.makedirs(dirname, exist_ok=True) + + if out_file: + with open(out_file, "wt") as f: + f.write(profiler.output_html()) + webbrowser.open(f"file://{os.path.realpath(out_file)}") diff --git a/activitysim/cli/benchmark.py b/activitysim/cli/benchmark.py index cbb1512d94..af46766aaf 100644 --- a/activitysim/cli/benchmark.py +++ b/activitysim/cli/benchmark.py @@ -37,18 +37,23 @@ "matrix": { "pyarrow": [], "numpy": [], + "scipy": ["1.7"], "openmatrix": [], - "pandas": [], + "pandas": ["1.3"], "pyyaml": [], "pytables": [], "toolz": [], "orca": [], "psutil": [], "requests": [], - "numba": [], - "coverage": [], - "pytest": [], + "numba": ["0.54"], "cytoolz": [], + "zarr": [], + "xarray": [], + "filelock": [], + "dask": [], + "networkx": [], + "sharrow": [], }, # The directory (relative to the current directory) to cache the Python # environments in. If not provided, defaults to "env" diff --git a/activitysim/cli/cli.py b/activitysim/cli/cli.py index 23d07812a1..de95e78ffa 100644 --- a/activitysim/cli/cli.py +++ b/activitysim/cli/cli.py @@ -25,4 +25,4 @@ def add_subcommand(self, name, args_func, exec_func, description): def execute(self): args = self.parser.parse_args() - args.afunc(args) + return args.afunc(args) diff --git a/activitysim/cli/create.py b/activitysim/cli/create.py index 85187c63e6..ec545d3787 100644 --- a/activitysim/cli/create.py +++ b/activitysim/cli/create.py @@ -1,8 +1,10 @@ import glob import hashlib +import logging import os import shutil import sys +from pathlib import Path import pkg_resources import requests @@ -53,6 +55,12 @@ def add_create_args(parser): help="path to new project directory (default: %(default)s)", ) + parser.add_argument( + "--link", + action="store_true", + help="cache and reuse downloaded files via symlinking", + ) + def create(args): """ @@ -66,12 +74,15 @@ def create(args): if args.list: list_examples() - sys.exit(0) + return 0 if args.example: - - get_example(args.example, args.destination) - sys.exit(0) + try: + get_example(args.example, args.destination, link=args.link) + except Exception: + logging.getLogger().exception("failure in activitysim create") + return 101 + return 0 def list_examples(): @@ -86,7 +97,9 @@ def list_examples(): return ret -def get_example(example_name, destination, benchmarking=False): +def get_example( + example_name, destination, benchmarking=False, optimize=True, link=True +): """ Copy project data to user-specified directory. @@ -95,8 +108,6 @@ def get_example(example_name, destination, benchmarking=False): `include` field which is a list of files/folders to include in the copied example. - - Parameters ---------- @@ -107,6 +118,13 @@ def get_example(example_name, destination, benchmarking=False): will be copied into a subdirectory with the same name as the example benchmarking: bool + optimize: bool + link: bool or path-like + Files downloaded via http pointers will be cached in + this location. If a path is not given but just a truthy + value, then a cache directory is created using in a location + selected by the appdirs library (or, if not installed, + linking is skipped.) """ if example_name not in EXAMPLES: sys.exit(f"error: could not find example '{example_name}'") @@ -137,7 +155,7 @@ def get_example(example_name, destination, benchmarking=False): sha256 = None if assets.startswith("http"): - download_asset(assets, target_path, sha256) + download_asset(assets, target_path, sha256, link=link) else: for asset_path in glob.glob(_example_path(assets)): @@ -145,6 +163,19 @@ def get_example(example_name, destination, benchmarking=False): print(f"copied! new project files are in {os.path.abspath(dest_path)}") + if optimize: + optimize_func_names = example.get("optimize", None) + if isinstance(optimize_func_names, str): + optimize_func_names = [optimize_func_names] + if optimize_func_names: + from ..examples import optimize_example_data + + for optimize_func_name in optimize_func_names: + getattr( + optimize_example_data, + optimize_func_name, + )(os.path.abspath(dest_path)) + instructions = example.get("instructions") if instructions: print(instructions) @@ -153,6 +184,7 @@ def get_example(example_name, destination, benchmarking=False): def copy_asset(asset_path, target_path, dirs_exist_ok=False): print(f"copying {os.path.basename(asset_path)} ...") + sys.stdout.flush() if os.path.isdir(asset_path): target_path = os.path.join(target_path, os.path.basename(asset_path)) shutil.copytree(asset_path, target_path, dirs_exist_ok=dirs_exist_ok) @@ -164,44 +196,89 @@ def copy_asset(asset_path, target_path, dirs_exist_ok=False): shutil.copy(asset_path, target_path) -def download_asset(url, target_path, sha256=None): +def download_asset(url, target_path, sha256=None, link=True): + if link: + if not isinstance(link, (str, Path)): + try: + import appdirs + except ImportError: + link = False + else: + link = appdirs.user_data_dir("ActivitySim") + original_target_path = target_path + target_path = os.path.join(link, target_path) os.makedirs(os.path.dirname(target_path), exist_ok=True) if url.endswith(".gz") and not target_path.endswith(".gz"): target_path_dl = target_path + ".gz" else: target_path_dl = target_path + download = True if sha256 and os.path.isfile(target_path): computed_sha256 = sha256_checksum(target_path) if sha256 == computed_sha256: print(f"not re-downloading existing {os.path.basename(target_path)} ...") - return + download = False else: print(f"re-downloading existing {os.path.basename(target_path)} ...") print(f" expected checksum {sha256}") print(f" computed checksum {computed_sha256}") else: print(f"downloading {os.path.basename(target_path)} ...") - with requests.get(url, stream=True) as r: - r.raise_for_status() - with open(target_path_dl, "wb") as f: - for chunk in r.iter_content(chunk_size=None): - f.write(chunk) - if target_path_dl != target_path: - import gzip - - with gzip.open(target_path_dl, "rb") as f_in: - with open(target_path, "wb") as f_out: - shutil.copyfileobj(f_in, f_out) - os.remove(target_path_dl) - computed_sha256 = sha256_checksum(target_path) - if sha256 and sha256 != computed_sha256: - raise ValueError( - f"downloaded {os.path.basename(target_path)} has incorrect checksum\n" - f" expected checksum {sha256}\n" - f" computed checksum {computed_sha256}" + sys.stdout.flush() + if download: + with requests.get(url, stream=True) as r: + r.raise_for_status() + print(f"| as {target_path_dl}") + with open(target_path_dl, "wb") as f: + for chunk in r.iter_content(chunk_size=None): + f.write(chunk) + if target_path_dl != target_path: + import gzip + + with gzip.open(target_path_dl, "rb") as f_in: + with open(target_path, "wb") as f_out: + print(f"| unzip to {target_path}") + shutil.copyfileobj(f_in, f_out) + os.remove(target_path_dl) + computed_sha256 = sha256_checksum(target_path) + if sha256 and sha256 != computed_sha256: + raise ValueError( + f"downloaded {os.path.basename(target_path)} has incorrect checksum\n" + f" expected checksum {sha256}\n" + f" computed checksum {computed_sha256}" + ) + elif not sha256: + print(f"| computed checksum {computed_sha256}") + if link: + os.makedirs( + os.path.dirname(os.path.normpath(original_target_path)), + exist_ok=True, ) - elif not sha256: - print(f" computed checksum {computed_sha256}") + + # check if the original_target_path exists and if so check if it is the correct file + if os.path.isfile(os.path.normpath(original_target_path)): + if sha256 is None: + sha256 = sha256_checksum(os.path.normpath(target_path)) + existing_sha256 = sha256_checksum(os.path.normpath(original_target_path)) + if existing_sha256 != sha256: + os.unlink(os.path.normpath(original_target_path)) + + # if the original_target_path exists now it is the correct file, keep it + if not os.path.isfile(os.path.normpath(original_target_path)): + try: + os.symlink( + os.path.normpath(target_path), + os.path.normpath(original_target_path), + ) + except OSError: + # permission errors likely foil symlinking on windows + shutil.copy( + os.path.normpath(target_path), + os.path.normpath(original_target_path), + ) + print(f"| copied to {os.path.normpath(original_target_path)}") + else: + print(f"| symlinked to {os.path.normpath(original_target_path)}") def sha256_checksum(filename, block_size=65536): @@ -210,3 +287,18 @@ def sha256_checksum(filename, block_size=65536): for block in iter(lambda: f.read(block_size), b""): sha256.update(block) return sha256.hexdigest() + + +def display_sha256_checksums(directory=None): + print("SHA 256 CHECKSUMS") + if directory is None: + if len(sys.argv) > 1: + directory = sys.argv[1] + else: + directory = os.getcwd() + print(f" in {directory}") + for dirpath, dirnames, filenames in os.walk(directory): + print(f"- in {dirpath}") + for filename in filenames: + f = os.path.join(dirpath, filename) + print(f"= {sha256_checksum(f)} = {f}") diff --git a/activitysim/cli/main.py b/activitysim/cli/main.py index 92988e863a..3805779290 100644 --- a/activitysim/cli/main.py +++ b/activitysim/cli/main.py @@ -2,18 +2,9 @@ import sys -def main(): - - # set all these before we import numpy or any other math library - if len(sys.argv) > 1 and sys.argv[1] == "benchmark": - os.environ["MKL_NUM_THREADS"] = "1" - os.environ["OMP_NUM_THREADS"] = "1" - os.environ["OPENBLAS_NUM_THREADS"] = "1" - os.environ["NUMBA_NUM_THREADS"] = "1" - os.environ["VECLIB_MAXIMUM_THREADS"] = "1" - os.environ["NUMEXPR_NUM_THREADS"] = "1" +def prog(): - from activitysim import __doc__, __version__ + from activitysim import __doc__, __version__, workflows from activitysim.cli import CLI, benchmark, create, run asim = CLI(version=__version__, description=__doc__) @@ -35,4 +26,41 @@ def main(): exec_func=benchmark.benchmark, description=benchmark.benchmark.__doc__, ) - sys.exit(asim.execute()) + asim.add_subcommand( + name="workflow", + args_func=lambda x: None, + exec_func=workflows.main, + description=workflows.main.__doc__, + ) + return asim + + +def main(): + + # set all these before we import numpy or any other math library + if len(sys.argv) > 1 and sys.argv[1] == "benchmark": + os.environ["MKL_NUM_THREADS"] = "1" + os.environ["OMP_NUM_THREADS"] = "1" + os.environ["OPENBLAS_NUM_THREADS"] = "1" + os.environ["NUMBA_NUM_THREADS"] = "1" + os.environ["VECLIB_MAXIMUM_THREADS"] = "1" + os.environ["NUMEXPR_NUM_THREADS"] = "1" + + asim = prog() + try: + if len(sys.argv) >= 2 and sys.argv[1] == "workflow": + from activitysim import workflows + + sys.exit(workflows.main(sys.argv[2:])) + else: + sys.exit(asim.execute()) + except Exception: + # if we are in the debugger, re-raise the error instead of exiting + if sys.gettrace() is not None: + raise + sys.exit(99) + + +def parser(): + asim = prog() + return asim.parser diff --git a/activitysim/cli/run.py b/activitysim/cli/run.py index d5c4649505..696048d0cf 100644 --- a/activitysim/cli/run.py +++ b/activitysim/cli/run.py @@ -13,7 +13,13 @@ logger = logging.getLogger(__name__) -INJECTABLES = ["data_dir", "configs_dir", "output_dir", "settings_file_name"] +INJECTABLES = [ + "data_dir", + "configs_dir", + "output_dir", + "settings_file_name", + "imported_extensions", +] def add_run_args(parser, multiprocess=True): @@ -64,6 +70,22 @@ def add_run_args(parser, multiprocess=True): parser.add_argument( "--households_sample_size", type=int, metavar="N", help="households sample size" ) + parser.add_argument( + "--fast", + action="store_true", + help="Do not limit process to one thread. " + "Can make single process runs faster, " + "but will cause thrashing on MP runs.", + ) + parser.add_argument( + "-e", + "--ext", + type=str, + action="append", + metavar="PATH", + help="Package of extension modules to load. Use of this option is not " + "generally secure.", + ) if multiprocess: parser.add_argument( @@ -87,7 +109,7 @@ def validate_injectable(name): # injectable is missing, meaning is hasn't been explicitly set # and defaults cannot be found. sys.exit( - "Error: please specify either a --working_dir " + f"Error({name}): please specify either a --working_dir " "containing 'configs', 'data', and 'output' folders " "or all three of --config, --data, and --output" ) @@ -111,6 +133,25 @@ def inject_arg(name, value, cache=False): # 'configs', 'data', and 'output' folders by default os.chdir(args.working_dir) + if args.ext: + import importlib + + for e in args.ext: + basepath, extpath = os.path.split(e) + if not basepath: + basepath = "." + sys.path.insert(0, basepath) + try: + importlib.import_module(e) + except ImportError as err: + logger.exception("ImportError") + raise + finally: + del sys.path[0] + inject_arg("imported_extensions", args.ext) + else: + inject_arg("imported_extensions", ()) + # settings_file_name should be cached or else it gets squashed by config.py if args.settings_file: inject_arg("settings_file_name", args.settings_file, cache=True) @@ -159,8 +200,16 @@ def cleanup_output_files(): tracing.delete_trace_files() + csv_ignore = [] + if config.setting("memory_profile", False): + # memory profiling is opened potentially before `cleanup_output_files` + # is called, but we want to leave any (newly created) memory profiling + # log files that may have just been created. + mem_prof_log = config.log_file_path("memory_profile.csv") + csv_ignore.append(mem_prof_log) + tracing.delete_output_files("h5") - tracing.delete_output_files("csv") + tracing.delete_output_files("csv", ignore=csv_ignore) tracing.delete_output_files("txt") tracing.delete_output_files("yaml") tracing.delete_output_files("prof") @@ -183,13 +232,27 @@ def run(args): # other callers (e.g. populationsim) will have to arrange to register their own steps and injectables # (presumably) in a custom run_simulation.py instead of using the 'activitysim run' command if not inject.is_injectable("preload_injectables"): - from activitysim import ( # register abm steps and other abm-specific injectables - abm, - ) + # register abm steps and other abm-specific injectables + from activitysim import abm # noqa: F401 tracing.config_logger(basic=True) handle_standard_args(args) # possibly update injectables + if config.setting("rotate_logs", False): + config.rotate_log_directory() + + if config.setting("memory_profile", False) and not config.setting( + "multiprocess", False + ): + # Memory sidecar is only useful for single process runs + # multiprocess runs log memory usage without blocking in the controlling process. + mem_prof_log = config.log_file_path("memory_profile.csv") + from ..core.memory_sidecar import MemorySidecar + + memory_sidecar_process = MemorySidecar(mem_prof_log) + else: + memory_sidecar_process = None + # legacy support for run_list setting nested 'models' and 'resume_after' settings if config.setting("run_list"): warnings.warn( @@ -281,7 +344,11 @@ def run(args): else: logger.info("run single process simulation") - pipeline.run(models=config.setting("models"), resume_after=resume_after) + pipeline.run( + models=config.setting("models"), + resume_after=resume_after, + memory_sidecar_process=memory_sidecar_process, + ) if config.setting("cleanup_pipeline_after_run", False): pipeline.cleanup_pipeline() # has side effect of closing open pipeline @@ -298,14 +365,21 @@ def run(args): chunk.consolidate_logs() mem.consolidate_logs() + from ..core.flow import TimeLogger + + TimeLogger.aggregate_summary(logger) + tracing.print_elapsed_time("all models", t0) + if memory_sidecar_process: + memory_sidecar_process.stop() + return 0 if __name__ == "__main__": - from activitysim import abm # register injectables + from activitysim import abm # register injectables # noqa: F401 parser = argparse.ArgumentParser() add_run_args(parser) diff --git a/activitysim/cli/test/test_cli.py b/activitysim/cli/test/test_cli.py index 96c3575c0a..b8f0fc3842 100644 --- a/activitysim/cli/test/test_cli.py +++ b/activitysim/cli/test/test_cli.py @@ -71,7 +71,7 @@ def test_run(): cp = subprocess.run(["activitysim", "run"], capture_output=True) msg = ( - "Error: please specify either a --working_dir " + "please specify either a --working_dir " "containing 'configs', 'data', and 'output' " "folders or all three of --config, --data, and --output" ) diff --git a/activitysim/core/assign.py b/activitysim/core/assign.py index 4eb55ddc32..f8e73adc6b 100644 --- a/activitysim/core/assign.py +++ b/activitysim/core/assign.py @@ -7,7 +7,7 @@ import numpy as np import pandas as pd -from activitysim.core import chunk, config, inject, pipeline, util +from activitysim.core import chunk, config, pipeline, util logger = logging.getLogger(__name__) @@ -264,6 +264,41 @@ def to_series(x): variables = OrderedDict() temps = OrderedDict() + # draw required randomness in one slug + # TODO: generalize to all randomness, not just lognormals + n_randoms = 0 + for expression_idx in assignment_expressions.index: + expression = assignment_expressions.loc[expression_idx, "expression"] + if "rng.lognormal_for_df(df," in expression: + expression = expression.replace( + "rng.lognormal_for_df(df,", f"rng_lognormal(random_draws[{n_randoms}]," + ) + n_randoms += 1 + assignment_expressions.loc[expression_idx, "expression"] = expression + if n_randoms: + from activitysim.core import pipeline + + try: + random_draws = pipeline.get_rn_generator().normal_for_df( + df, broadcast=True, size=n_randoms + ) + except RuntimeError: + pass + else: + _locals_dict["random_draws"] = random_draws + + def rng_lognormal(random_draws, mu, sigma, broadcast=True, scale=False): + if scale: + x = 1 + ((sigma * sigma) / (mu * mu)) + mu = np.log(mu / (np.sqrt(x))) + sigma = np.sqrt(np.log(x)) + assert broadcast + return np.exp(random_draws * sigma + mu) + + _locals_dict["rng_lognormal"] = rng_lognormal + + sharrow_enabled = config.setting("sharrow", False) + # need to be able to identify which variables causes an error, which keeps # this from being expressed more parsimoniously @@ -319,6 +354,18 @@ def to_series(x): globals_dict = {} expr_values = to_series(eval(expression, globals_dict, _locals_dict)) + if ( + sharrow_enabled + and np.issubdtype(expr_values.dtype, np.floating) + and expr_values.dtype.itemsize < 4 + ): + # promote to float32, numba is not presently compatible with + # any float less than 32 (i.e., float16) + # see https://github.com/numba/numba/issues/4402 + # note this only applies to floats, signed and unsigned + # integers are readily supported down to 1 byte + expr_values = expr_values.astype(np.float32) + np.seterr(**save_err) np.seterrcall(saved_handler) diff --git a/activitysim/core/choosing.py b/activitysim/core/choosing.py new file mode 100644 index 0000000000..d32f254019 --- /dev/null +++ b/activitysim/core/choosing.py @@ -0,0 +1,202 @@ +import numpy as np +import pandas as pd +from numba import njit + + +@njit +def choice_maker(pr, rn, out=None): + if out is None: + out = np.empty(pr.shape[0], dtype=np.int32) + n_alts = pr.shape[1] + for row in range(pr.shape[0]): + z = rn[row] + for col in range(n_alts): + z = z - pr[row, col] + if z <= 0: + out[row] = col + break + else: + # rare condition, only if a random point is greater than 1 (a bug) + # or if the sum of probabilities is less than 1 and a random point + # is greater than that sum, which due to the limits of numerical + # precision can technically happen + max_pr = 0.0 + for col in range(n_alts): + if pr[row, col] > max_pr: + out[row] = col + max_pr = pr[row, col] + return out + + +@njit +def sample_choices_maker( + prob_array, + random_array, + alts_array, + out_choices=None, + out_choice_probs=None, +): + """ + Random sample of alternatives. + + Parameters + ---------- + prob_array : array of float, shape (n_choosers, n_alts) + random_array : array of float, shape (n_choosers, n_samples) + alts_array : array of int, shape (n_alts) + out_choices : array of int, shape (n_samples, n_choosers), optional + out_choice_probs : array of float, shape (n_samples, n_choosers), optional + + Returns + ------- + out_choices, out_choice_probs + """ + n_choosers = random_array.shape[0] + sample_size = random_array.shape[1] + n_alts = prob_array.shape[1] + if out_choices is None: + out_choices = np.empty((sample_size, n_choosers), dtype=np.int32) + if out_choice_probs is None: + out_choice_probs = np.empty((sample_size, n_choosers), dtype=np.float32) + + for c in range(n_choosers): + random_points = np.sort(random_array[c, :]) + a = 0 + s = 0 + z = 0.0 + for a in range(n_alts): + z += prob_array[c, a] + while s < sample_size and z > random_points[s]: + out_choices[s, c] = alts_array[a] + out_choice_probs[s, c] = prob_array[c, a] + s += 1 + if s >= sample_size: + break + if s < sample_size: + # rare condition, only if a random point is greater than 1 (a bug) + # or if the sum of probabilities is less than 1 and a random point + # is greater than that sum, which due to the limits of numerical + # precision can technically happen + a = n_alts - 1 + while prob_array[c, a] < 1e-30 and a > 0: + # slip back to the last choice with non-trivial prob + a -= 1 + while s < sample_size: + out_choices[s, c] = alts_array[a] + out_choice_probs[s, c] = prob_array[c, a] + s += 1 + + return out_choices, out_choice_probs + + +@njit +def _sample_choices_maker_preserve_ordering( + prob_array, + random_array, + alts_array, + out_choices=None, + out_choice_probs=None, +): + """ + Random sample of alternatives. + + Preserves the ordering of the random array, consistent with the legacy + ActivitySim implementation. + + Parameters + ---------- + prob_array : array of float, shape (n_choosers, n_alts) + random_array : array of float, shape (n_choosers, n_samples) + alts_array : array of int, shape (n_alts) + out_choices : array of int, shape (n_samples, n_choosers), optional + out_choice_probs : array of float, shape (n_samples, n_choosers), optional + + Returns + ------- + out_choices, out_choice_probs + """ + n_choosers = random_array.shape[0] + sample_size = random_array.shape[1] + n_alts = prob_array.shape[1] + if out_choices is None: + out_choices = np.empty((sample_size, n_choosers), dtype=np.int32) + if out_choice_probs is None: + out_choice_probs = np.empty((sample_size, n_choosers), dtype=np.float32) + + for c in range(n_choosers): + sorted_random = np.argsort(random_array[c, :]) + a = 0 + s = 0 + z = 0.0 + for a in range(n_alts): + z += prob_array[c, a] + while s < sample_size and z > random_array[c, sorted_random[s]]: + out_choices[sorted_random[s], c] = alts_array[a] + out_choice_probs[sorted_random[s], c] = prob_array[c, a] + s += 1 + if s >= sample_size: + break + if s < sample_size: + # rare condition, only if a random point is greater than 1 (a bug) + # or if the sum of probabilities is less than 1 and a random point + # is greater than that sum, which due to the limits of numerical + # precision can technically happen + a = n_alts - 1 + while prob_array[c, a] < 1e-30 and a > 0: + # slip back to the last choice with non-trivial prob + a -= 1 + while s < sample_size: + out_choices[sorted_random[s], c] = alts_array[a] + out_choice_probs[sorted_random[s], c] = prob_array[c, a] + s += 1 + + return out_choices, out_choice_probs + + +def sample_choices_maker_preserve_ordering( + prob_array, + random_array, + alts_array, + out_choices=None, + out_choice_probs=None, +): + """ + Make sample choices. + + Parameters + ---------- + prob_array : array[float] + random_array : array[float] + alts_array : array + out_choices : array[alts_array.dtype], optional + out_choice_probs : array[float], optional + + Returns + ------- + out_choices : array[alts_array.dtype] + out_choice_probs : array[float] + """ + if alts_array.dtype.kind != "i": + # when the alternatives array is not integers (which is unusual, but + # does happen in the OD choice model) we need to choose integers in + # numba and then convert those back to whatever dtype the alternatives are + out_choices_, out_choice_probs = _sample_choices_maker_preserve_ordering( + prob_array, + random_array, + np.arange(alts_array.size), + out_choices=None, + out_choice_probs=out_choice_probs, + ) + if out_choices is not None: + out_choices[:] = alts_array[out_choices_] + else: + out_choices = alts_array[out_choices_] + return out_choices, out_choice_probs + + return _sample_choices_maker_preserve_ordering( + prob_array, + random_array, + alts_array, + out_choices=out_choices, + out_choice_probs=out_choice_probs, + ) diff --git a/activitysim/core/chunk.py b/activitysim/core/chunk.py index 547c8f064d..6634b479d8 100644 --- a/activitysim/core/chunk.py +++ b/activitysim/core/chunk.py @@ -8,10 +8,12 @@ import multiprocessing import os import threading +import warnings from contextlib import contextmanager import numpy as np import pandas as pd +import xarray as xr from . import config, mem, tracing, util from .util import GB @@ -539,10 +541,10 @@ def size_it(df): elif isinstance(df, pd.DataFrame): elements = util.iprod(df.shape) bytes = 0 if not elements else df.memory_usage(index=True).sum() - elif isinstance(df, np.ndarray): + elif isinstance(df, (np.ndarray, xr.DataArray)): elements = util.iprod(df.shape) bytes = df.nbytes - elif isinstance(df, list): + elif isinstance(df, (list, tuple)): # dict of series, dataframe, or ndarray (e.g. assign assign_variables target and temp dicts) elements = 0 bytes = 0 @@ -550,7 +552,7 @@ def size_it(df): e, b = size_it(v) elements += e bytes += b - elif isinstance(df, dict): + elif isinstance(df, (dict, xr.Dataset)): # dict of series, dataframe, or ndarray (e.g. assign assign_variables target and temp dicts) elements = 0 bytes = 0 @@ -581,6 +583,8 @@ def size_it(df): shape = f"list({[x.shape for x in df]})" elif isinstance(df, dict): shape = f"dict({[v.shape for v in df.values()]})" + elif isinstance(df, xr.Dataset): + shape = df.dims else: shape = df.shape @@ -846,7 +850,11 @@ def initial_rows_per_chunk(self): ) estimated_number_of_chunks = None - assert chunk_training_mode() != MODE_PRODUCTION + if chunk_training_mode() == MODE_PRODUCTION: + warnings.warn( + "ActivitySim is running with a chunk_training_mode of " + f"'production' but initial_row_size is zero in {self.trace_label}" + ) # cum_rows is out of phase with cum_overhead # since we won't know observed_chunk_size until AFTER yielding the chunk @@ -1039,6 +1047,10 @@ def chunk_log(trace_label, chunk_tag=None, base=False): # a ChunkSizer class object without actually chunking. This # avoids breaking the assertion below. + if chunk_training_mode() == MODE_CHUNKLESS: + yield + return + assert base == (len(CHUNK_SIZERS) == 0) trace_label = f"{trace_label}.chunk_log" @@ -1073,6 +1085,14 @@ def adaptive_chunked_choosers(choosers, chunk_size, trace_label, chunk_tag=None) # generator to iterate over choosers + if chunk_training_mode() == MODE_CHUNKLESS: + # The adaptive chunking logic is expensive and sometimes results + # in needless data copying. So we short circuit it entirely + # when chunking is disabled. + logger.info(f"Running chunkless with {len(choosers)} choosers") + yield 0, choosers, trace_label + return + chunk_tag = chunk_tag or trace_label num_choosers = len(choosers.index) @@ -1154,6 +1174,18 @@ def adaptive_chunked_choosers_and_alts( chunk of alternatives for chooser chunk """ + if chunk_training_mode() == MODE_CHUNKLESS: + # The adaptive chunking logic is expensive and sometimes results + # in needless data copying. So we short circuit it entirely + # when chunking is disabled. + logger.info(f"Running chunkless with {len(choosers)} choosers") + yield 0, choosers, alternatives, trace_label + return + + check_assertions = False + # set to True if debugging is needed; there are many expensive assertions + # to check data quality in here + chunk_tag = chunk_tag or trace_label num_choosers = len(choosers.index) @@ -1161,18 +1193,17 @@ def adaptive_chunked_choosers_and_alts( assert num_choosers > 0 # alternatives index should match choosers (except with duplicate repeating alt rows) - assert choosers.index.equals( - alternatives.index[~alternatives.index.duplicated(keep="first")] - ) + if check_assertions: + assert choosers.index.equals( + alternatives.index[~alternatives.index.duplicated(keep="first")] + ) last_repeat = alternatives.index != np.roll(alternatives.index, -1) - assert (num_choosers == 1) or choosers.index.equals(alternatives.index[last_repeat]) - assert ( - "pick_count" in alternatives.columns - or choosers.index.name == alternatives.index.name - ) - assert choosers.index.name == alternatives.index.name + if check_assertions: + assert (num_choosers == 1) or choosers.index.equals( + alternatives.index[last_repeat] + ) logger.info( f"{trace_label} Running adaptive_chunked_choosers_and_alts " @@ -1210,12 +1241,13 @@ def adaptive_chunked_choosers_and_alts( alt_end = alt_chunk_ends[offset + rows_per_chunk] alternative_chunk = alternatives[alt_offset:alt_end] - assert len(chooser_chunk.index) == len( - np.unique(alternative_chunk.index.values) - ) - assert ( - chooser_chunk.index == np.unique(alternative_chunk.index.values) - ).all() + if check_assertions: + assert len(chooser_chunk.index) == len( + np.unique(alternative_chunk.index.values) + ) + assert ( + chooser_chunk.index == np.unique(alternative_chunk.index.values) + ).all() logger.info( f"Running chunk {i} of {estimated_number_of_chunks or '?'} " @@ -1245,6 +1277,14 @@ def adaptive_chunked_choosers_by_chunk_id( # all have to be included in the same chunk) # FIXME - we pathologically know name of chunk_id col in households table + if chunk_training_mode() == MODE_CHUNKLESS: + # The adaptive chunking logic is expensive and sometimes results + # in needless data copying. So we short circuit it entirely + # when chunking is disabled. + logger.info(f"Running chunkless with {len(choosers)} choosers") + yield 0, choosers, trace_label + return + chunk_tag = chunk_tag or trace_label num_choosers = choosers["chunk_id"].max() + 1 diff --git a/activitysim/core/cleaning.py b/activitysim/core/cleaning.py new file mode 100644 index 0000000000..f5fa153561 --- /dev/null +++ b/activitysim/core/cleaning.py @@ -0,0 +1,66 @@ +import numpy as np +import pandas as pd + +from . import inject + + +def recode_to_zero_based(values, mapping): + values = np.asarray(values) + zone_ids = pd.Index(mapping, dtype=np.int32) + if ( + zone_ids.is_monotonic_increasing + and zone_ids[-1] == len(zone_ids) + zone_ids[0] - 1 + ): + offset = zone_ids[0] + result = values - offset + else: + n = len(zone_ids) + remapper = dict(zip(zone_ids, pd.RangeIndex(n))) + if n < 128: + out_dtype = np.int8 + elif n < (1 << 15): + out_dtype = np.int16 + elif n < (1 << 31): + out_dtype = np.int32 + else: + out_dtype = np.int64 + result = np.fromiter((remapper.get(xi) for xi in values), out_dtype) + return result + + +def should_recode_based_on_table(tablename): + try: + base_df = inject.get_table(tablename).to_frame() + except (KeyError, RuntimeError): + # the basis table is missing, do not + return False + if base_df.index.name and f"_original_{base_df.index.name}" in base_df: + return True + return False + + +def recode_based_on_table(values, tablename): + try: + base_df = inject.get_table(tablename).to_frame() + except (KeyError, RuntimeError): + # the basis table is missing, do nothing + return values + if base_df.index.name and f"_original_{base_df.index.name}" in base_df: + source_ids = base_df[f"_original_{base_df.index.name}"] + if ( + isinstance(base_df.index, pd.RangeIndex) + and base_df.index.start == 0 + and base_df.index.step == 1 + ): + return recode_to_zero_based(values, source_ids) + elif ( + base_df.index.is_monotonic_increasing + and base_df.index[0] == 0 + and base_df.index[-1] == len(base_df) - 1 + ): + return recode_to_zero_based(values, source_ids) + else: + remapper = dict(zip(source_ids, base_df.index)) + return np.fromiter((remapper.get(xi) for xi in values), base_df.index.dtype) + else: + return values diff --git a/activitysim/core/config.py b/activitysim/core/config.py index 02d6fadea2..56237b1a62 100644 --- a/activitysim/core/config.py +++ b/activitysim/core/config.py @@ -108,6 +108,13 @@ def get_cache_dir(): os.mkdir(cache_dir) assert os.path.isdir(cache_dir) + # create a git-ignore in the cache dir if it does not exist. + # this helps prevent accidentally committing cache contents to git + gitignore = os.path.join(cache_dir, ".gitignore") + if not os.path.exists(gitignore): + with open(gitignore, "wt") as f: + f.write("/*") + return cache_dir @@ -248,18 +255,19 @@ def cascading_input_file_path( dir_paths = [dir_paths] if isinstance(dir_paths, str) else dir_paths file_path = None - for dir in dir_paths: - p = os.path.join(dir, file_name) - if os.path.isfile(p): - file_path = p - break + if file_name is not None: + for dir in dir_paths: + p = os.path.join(dir, file_name) + if os.path.isfile(p): + file_path = p + break - if allow_glob and len(glob.glob(p)) > 0: - file_path = p - break + if allow_glob and len(glob.glob(p)) > 0: + file_path = p + break if mandatory and not file_path: - raise RuntimeError( + raise FileNotFoundError( "file_path %s: file '%s' not in %s" % (dir_list_injectable_name, file_name, dir_paths) ) @@ -337,6 +345,20 @@ def output_file_path(file_name): return build_output_file_path(file_name, use_prefix=prefix) +def profiling_file_path(file_name): + + profile_dir = inject.get_injectable("profile_dir", None) + if profile_dir is None: + output_dir = inject.get_injectable("output_dir") + profile_dir = os.path.join( + output_dir, time.strftime("profiling--%Y-%m-%d--%H-%M-%S") + ) + os.makedirs(profile_dir, exist_ok=True) + inject.add_injectable("profile_dir", profile_dir) + + return os.path.join(profile_dir, file_name) + + def trace_file_path(file_name): output_dir = inject.get_injectable("output_dir") @@ -399,6 +421,31 @@ def open_log_file(file_name, mode, header=None, prefix=False): return f +def rotate_log_directory(): + + output_dir = inject.get_injectable("output_dir") + log_dir = os.path.join(output_dir, "log") + if not os.path.exists(log_dir): + return + + from datetime import datetime + from stat import ST_CTIME + + old_log_time = os.stat(log_dir)[ST_CTIME] + rotate_name = os.path.join( + output_dir, + datetime.fromtimestamp(old_log_time).strftime("log--%Y-%m-%d--%H-%M-%S"), + ) + try: + os.rename(log_dir, rotate_name) + except Exception as err: + # if Windows fights us due to permissions or whatever, + print(f"unable to rotate log file, {err!r}") + else: + # on successful rotate, create new empty log directory + os.makedirs(log_dir) + + def pipeline_file_path(file_name): prefix = inject.get_injectable("pipeline_file_prefix", None) @@ -504,7 +551,7 @@ def backfill_settings(settings, backfill): # essentially the current settings firle is an alias for the included file if len(s) > 1: logger.error( - f"'include_settings' must appear alone in settings file." + "'include_settings' must appear alone in settings file." ) additional_settings = list( set(s.keys()).difference({"include_settings"}) @@ -513,7 +560,7 @@ def backfill_settings(settings, backfill): f"Unexpected additional settings: {additional_settings}" ) raise RuntimeError( - f"'include_settings' must appear alone in settings file." + "'include_settings' must appear alone in settings file." ) logger.debug( @@ -628,6 +675,14 @@ def filter_warnings(): message="`np.object` is a deprecated alias", ) + # Numba triggers a DeprecationWarning from numpy about np.MachAr + warnings.filterwarnings( + "ignore", + category=DeprecationWarning, + module="numba", + message=".np.MachAr. is deprecated", + ) + # beginning pandas version 1.3, various places emit a PerformanceWarning that is # caught in the "strict" filter above, but which are currently unavoidable for complex models. # These warning are left as warnings as an invitation for future enhancement. diff --git a/activitysim/core/configuration.py b/activitysim/core/configuration.py index 3174653b25..f8a70394af 100644 --- a/activitysim/core/configuration.py +++ b/activitysim/core/configuration.py @@ -1,3 +1,4 @@ +from numbers import Number from typing import Union try: @@ -27,20 +28,165 @@ class InputTable(PydanticBase): """table column to use for the index""" rename_columns: dict[str, str] = None - """dictionary of column name mappings""" + """ + Dictionary of column name mappings. + + This allows for renaming data columns from the original names found in the + header of the input file itself, into the names used internally by + ActivitySim, in configuration and specification files. + """ + + recode_columns: dict[str, str] = None + """ + Dictionary of column recoding instructions. + + Certain columns of data, notably TAZ and MAZ id's, are more efficiently + stored as the index (offset) position of each value within a fixed array of + values, instead of as the value itself. To recode a column into this offset + format, give the value "zero-based" for that column name. This will replace + the named column with a RangeIndex, starting from zero, and will create a + lookup column of the original values, which is not used by ActivitySim other + than to recode other related variables, or to reconstitute the original + labels for a final output table. This zero-based recoding is typically + done for the `zone_id` field in the land_use table, but might also be done + elsewhere. + + Alternatively, for columns that contain *references* to recoded data, give + the recode instruction as "tablename.fieldname" (often, "land_use.zone_id"). + This will trigger a remapping of each value according to the stored lookup + table for the original values, transforming the values in other columns to + be consistent with the recoded zero-based values. For example, if the + `zone_id` field in the land_use table has been recoded to be zero-based, + then the home_zone_id in the households table needs to be recoded to match. + + Note that recoding is done after renaming, so the key values in this mapping + should correspond to the internally used names and not the original column + names that appear in the input file (if they have been renamed). + """ keep_columns: list[str] = None """ - columns to keep once read in to memory. + Columns to keep once read in to memory. Save only the columns needed for modeling or analysis to save on memory - and file I/O + and file I/O. If not given, all columns in the input file will be read + and retained. """ h5_tablename: str = None """table name if reading from HDF5 and different from `tablename`""" +class OutputTable(PydanticBase): + tablename: str + """The name of the pipeline table to write out.""" + + decode_columns: dict[str, str] = None + """ + A mapping indicating columns to decode when writing out results. + + Column decoding is the inverse of column recoding, such that the original + mapped values are restored. For example, if TAZ ID's in the zone_id column + of the land_use table have been recoded to zero-based, then any output + column that gives a TAZ (i.e., household home zones, work or school + locations, trip or tour origins and destinations) can and probably should be + decoded from zero-based back into nominal TAZ ID's. If every value in the + column is to be decoded, simply give the decode instruction in the same + manner as the recode instruction, "tablename.fieldname" (often, + "land_use.zone_id"). + + For some columns, like work or school locations, only non-negative values + should be decoded, as negative values indicate an absence of choice. In + these cases, the "tablename.fieldname" can be prefixed with a "nonnegative" + filter, seperated by a pipe character (e.g. "nonnegative | land_use.zone_id"). + """ + + +class OutputTables(PydanticBase): + """Instructions on how to write out final pipeline tables.""" + + h5_store: bool = False + """Write tables into a single HDF5 store instead of individual CSVs.""" + + action: str + """Whether to 'include' or 'skip' the enumerated tables in `tables`.""" + + prefix: str = "final_" + """This prefix is added to the filename of each output table.""" + + sort: bool = False + """Sort output in each table consistent with well defined column names.""" + + tables: list[Union[str, OutputTable]] = None + """ + A list of pipeline tables to include or to skip when writing outputs. + + If `action` is 'skip', the values in this list must all be simple + strings giving the names of tables to skip. Also, no decoding will be + applied to any output tables in this case. + + If `action` is 'include', the values in this list can be either simple + strings giving the names of tables to include, or :class:`OutputTable` + definitions giving a name and decoding instructions. + + If omitted, the all tables are written out and no decoding will be + applied to any output tables. + """ + + +class MultiprocessStepSlice(PydanticBase): + """Instructions on how to slice tables for each subprocess.""" + + tables: list[str] + """ + The names of tables that are to be sliced for multiprocessing. + + The index of the first table in the 'tables' list is the primary_slicer. + Any other tables listed are dependent tables with either ref_cols to the + primary_slicer or with the same index (i.e. having an index with the same + name). This cascades, so any tables dependent on the primary_table can in + turn have dependent tables that will be sliced by index or ref_col. + + For instance, if the primary_slicer is households, then persons can be + sliced because it has a ref_col to (column with the same same name as) the + household table index. And the tours table can be sliced since it has a + ref_col to persons. Tables can also be sliced by index. For instance the + person_windows table can be sliced because it has an index with the same + names as the persons table. + """ + + exclude: Union[bool, str, list[str]] + """ + Optional list of tables not to slice even if they have a sliceable index name. + + Or set to `True` or "*" to exclude all tables not explicitly listed in + `tables`. + """ + + +class MultiprocessStep(PydanticBase): + """ + A contiguous group of model components that are multiprocessed together. + """ + + name: str + """A descriptive name for this multiprocessing step.""" + + begin: str + """The first component that is part of this multiprocessing step.""" + + num_processes: int = None + """ + The number of processes to use in this multiprocessing step. + + If not provided, the default overall number of processes set in the main + settings file is used. + """ + + slice: MultiprocessStepSlice = None + """Instructions on how to slice tables for each subprocess.""" + + class Settings(PydanticBase): """ The overall settings for the ActivitySim model system. @@ -61,6 +207,20 @@ class Settings(PydanticBase): See :ref:`model_steps` for more details about each step. """ + multiprocess: bool = False + """Enable multiprocessing for this model.""" + + num_processes: int = None + """ + If running in multiprocessing mode, use this number of processes by default. + + If not given or set to 0, the number of processes to use is set to + half the number of available CPU cores, plus 1. + """ + + multiprocess_steps: list[MultiprocessStep] + """A list of multiprocess steps.""" + resume_after: str = None """to resume running the data pipeline after the last successful checkpoint""" @@ -146,7 +306,7 @@ class Settings(PydanticBase): use_shadow_pricing: bool = False """turn shadow_pricing on and off for work and school location""" - output_tables: list[str] = None + output_tables: OutputTables = None """list of output tables to write to CSV or HDF5""" want_dest_choice_sample_tables: bool = False @@ -160,6 +320,9 @@ class Settings(PydanticBase): single-checkpoint pipeline file, and deleting any subprocess pipelines. """ + cleanup_trace_files_on_resume: bool = False + """Clean all trace files when restarting a model from a checkpoint.""" + sharrow: Union[bool, str] = False """ Set the sharrow operating mode. @@ -179,6 +342,130 @@ class Settings(PydanticBase): mode of operation, but useful for development and debugging. """ + disable_zarr: bool = False + """ + Disable the use of zarr format skims. + + .. versionadded:: 1.2 + + By default, if sharrow is enabled (any setting other than false), ActivitySim + currently loads data from zarr format skims if a zarr location is provided, + and data is found there. If no data is found there, then original OMX skim + data is loaded, any transformations or encodings are applied, and then this + data is written out to a zarr file at that location. Setting this option to + True will disable the use of zarr. + """ + + instrument: bool = False + """ + Use `pyinstrument` to profile component performance. + + .. versionadded:: 1.2 + + This is generally a developer-only feature and not needed for regular usage + of ActivitySim. + + Use of this setting to enable statistical profiling of ActivitySim code, + using the `pyinstrument` library (an optional dependency which must also be + installed). A separate profiling session is triggered for each model + component. See the pyinstrument + `documentation `__ + for a description of how this tool works. + + When activated, a "profiling--\\*" directory is created in the output directory + of the model, tagged with the date and time of the profiling run. Profile + output is always tagged like this and never overwrites previous profiling + outputs, facilitating serial comparisons of runtimes in response to code or + configuration changes. + """ + + memory_profile: bool = False + """ + Generate a memory profile by sampling memory usage from a secondary process. + + .. versionadded:: 1.2 + + This is generally a developer-only feature and not needed for regular usage + of ActivitySim. + + Using this feature will open a secondary process, whose only job is to poll + memory usage for the main ActivitySim process. The usage is logged to a file + with time stamps, so it can be cross-referenced against ActivitySim logs to + identify what parts of the code are using RAM. The profiling is done from + a separate process to avoid the profiler itself from significantly slowing + the main model core, or (more importantly) generating memory usage on its + own that pollutes the collected data. + """ + + benchmarking: bool = False + """ + Flag this model run as a benchmarking run. + + .. versionadded:: 1.1 + + This is generally a developer-only feature and not needed for regular usage + of ActivitySim. + + By flagging a model run as a benchmark, certain operations of the model are + altered, to ensure valid benchmark readings. For example, in regular + operation, data such as skims are loaded on-demand within the first model + component that needs them. With benchmarking enabled, all data are always + pre-loaded before any component is run, to ensure that recorded times are + the runtime of the component itself, and not data I/O operations that are + neither integral to that component nor necessarily stable over replication. + """ + + write_raw_tables: bool = False + """ + Dump input tables back to disk immediately after loading them. + + This is generally a developer-only feature and not needed for regular usage + of ActivitySim. + + The data tables are written out before any annotation steps, but after + initial processing (renaming, filtering columns, recoding). + """ + + disable_destination_sampling: bool = False + + want_dest_choice_presampling: bool = False + + testing_fail_trip_destination: bool = False + + fail_fast: bool = False + + rotate_logs: bool = False + + offset_preprocessing: bool = False + """ + Flag to indicate whether offset preprocessing has already been done. + + .. versionadded:: 1.2 + + This flag is generally set automatically within ActivitySim during a run, + and not be a user ahead of time. The ability to do so is provided as a + developer-only feature for testing and development. + """ + + recode_pipeline_columns: bool = True + """ + Apply recoding instructions on input and final output for pipeline tables. + + .. versionadded:: 1.2 + + Recoding instructions can be provided in individual + :py:attr:`InputTable.recode_columns` and :py:attr:`OutputTable.decode_columns` + settings. This global setting permits disabling all recoding processes + simultaneously. + + .. warning:: + + Disabling recoding is fine in legacy mode but it is generally not + compatible with using :py:attr:`Settings.sharrow`. + """ + + keep_mem_logs: bool = False + class ZarrDigitalEncoding(PydanticBase): """Digital encoding instructions for skim tables. @@ -186,20 +473,85 @@ class ZarrDigitalEncoding(PydanticBase): .. versionadded:: 1.2 """ - regex: str + name: str = None + """The name of an individual matrix skim to encode. + + Use this setting to encode specific individual skims. To encode + a group of related skims with the same encoding, or together with + a joint encoding, use `regex` instead. You cannot specify both + `name` and `regex` at the same time. + """ + + regex: str = None """A regular expression for matching skim matrix names. All skims with names that match under typical regular expression rules - for Python will be processed together. + for Python will be processed using the rules defined in this + ZarrDigitalEncoding instruction. To encode one specific skim, + use `name` instead. You cannot specify both `name` and `regex` at + the same time. """ - joint_dict: str + joint_dict: str = None """The name of the joint dictionary for this group. This must be a unique name for this set of skims, and a new array will be added to the Dataset with this name. It will be an integer- type array indicating the position of each element in the jointly - encoded dictionary.""" + encoded dictionary. + + If the `joint_dict` name is given, then all other instructions in this + ZarrDigitalEncoding are ignored, except `regex`. + """ + + missing_value: Number = None + """ + Use this value to indicate "missing" values. + + For float variables, it is possible to use NaN to represent missing values, + but other data types do not have a native missing value, so it will need + to be given explicitly. + """ + + bitwidth: int = None + """Number of bits to use in encoded integers, either 8, 16 or 32.""" + + min_value: Number = None + """ + Explicitly give the minimum value represented in the array. + If not given, it is inferred from the data. It is useful to give + these values if the array does not necessarily include all the values that + might need to be inserted later. + """ + + max_value: Number = None + """ + Explicitly give the maximum value represented in the array. + If not given, it is inferred from the data. It is useful to give + these values if the array does not necessarily include all the values that + might need to be inserted later. + """ + + scale: Number = None + """ + An explicitly defined scaling factor. + + The scaling factor can be inferred from the min and max values if not provided. + """ + + offset: Number = None + """ + An explicitly defined offset factor. + + The offset factor can be inferred from the min and max values if not provided. + """ + + by_dict: int = None + """ + Encode by dictionary, using a bitwidth from {8, 16, 32}. + + If given, all arguments other settings for this data are ignored. + """ class TAZ_Settings(PydanticBase): @@ -232,6 +584,10 @@ class TAZ_Settings(PydanticBase): A list of encodings to apply before saving skims in ZARR format. .. versionadded:: 1.2 + + Digital encodings transform how data is stored in memory and on disk, + potentially reducing storage requirements without fundamentally changing + the underlying data. """ diff --git a/activitysim/core/expressions.py b/activitysim/core/expressions.py index a64f66523f..90a75e1379 100644 --- a/activitysim/core/expressions.py +++ b/activitysim/core/expressions.py @@ -84,12 +84,10 @@ def compute_columns(df, model_settings, locals_dict={}, trace_label=None): _locals_dict.update(tables) # FIXME a number of asim model preprocessors want skim_dict - should they request it in model_settings.TABLES? - _locals_dict.update( - { - # 'los': inject.get_injectable('network_los', None), - "skim_dict": inject.get_injectable("skim_dict", None), - } - ) + if config.setting("sharrow", False): + _locals_dict["skim_dict"] = inject.get_injectable("skim_dataset_dict", None) + else: + _locals_dict["skim_dict"] = inject.get_injectable("skim_dict", None) results, trace_results, trace_assigned_locals = assign.assign_variables( expressions_spec, df, _locals_dict, trace_rows=tracing.trace_targets(df) diff --git a/activitysim/core/fast_mapping.py b/activitysim/core/fast_mapping.py new file mode 100644 index 0000000000..5c3a4fd109 --- /dev/null +++ b/activitysim/core/fast_mapping.py @@ -0,0 +1,62 @@ +import numba as nb +import numpy as np +import pandas as pd + + +@nb.njit +def _fast_map(fm, target, dtype=np.int32): + out = np.zeros(len(target), dtype=dtype) + for n in range(target.size): + out[n] = fm[target[n]] + return out + + +class FastMapping: + def __init__(self, source, to_range=np.int64): + if isinstance(source, pd.Series): + m = nb.typed.Dict.empty( + key_type=nb.from_dtype(source.index.dtype), + value_type=nb.from_dtype(source.dtype), + ) + for k, v in source.items(): + m[k] = v + self._in_dtype = source.index.dtype + self._out_dtype = source.dtype + self._mapper = m + elif to_range: + m = nb.typed.Dict.empty( + key_type=nb.from_dtype(source.dtype), + value_type=nb.from_dtype(to_range), + ) + for v, k in enumerate(source): + m[k] = v + self._in_dtype = source.dtype + self._out_dtype = to_range + self._mapper = m + else: + raise ValueError("invalid input") + + def __len__(self): + return len(self._mapper) + + def __contains__(self, item): + return item in self._mapper + + def __getitem__(self, item): + return self._mapper[item] + + def apply_to(self, target): + if isinstance(target, pd.Series): + return pd.Series( + _fast_map( + self._mapper, + target.astype(self._in_dtype).to_numpy(), + dtype=self._out_dtype, + ), + index=target.index, + ) + return _fast_map( + self._mapper, + np.asarray(target, dtype=self._in_dtype), + dtype=self._out_dtype, + ) diff --git a/activitysim/core/flow.py b/activitysim/core/flow.py new file mode 100644 index 0000000000..8396e8ccba --- /dev/null +++ b/activitysim/core/flow.py @@ -0,0 +1,741 @@ +import contextlib +import glob +import logging +import os +import time +from datetime import timedelta +from numbers import Number +from stat import ST_MTIME + +import numpy as np +import pandas as pd + +from .. import __version__ +from ..core import tracing +from . import config, inject +from .simulate_consts import SPEC_EXPRESSION_NAME, SPEC_LABEL_NAME +from .timetable import ( + sharrow_tt_adjacent_window_after, + sharrow_tt_adjacent_window_before, + sharrow_tt_max_time_block_available, + sharrow_tt_previous_tour_begins, + sharrow_tt_previous_tour_ends, + sharrow_tt_remaining_periods_available, +) + +try: + import sharrow as sh +except ModuleNotFoundError: + sh = None + + +logger = logging.getLogger(__name__) + +_FLOWS = {} + +if os.environ.get("TRAVIS") == "true": + # The multithreaded dask scheduler causes problems on travis. + # Here, we detect if this code is running on Travis, and if so, we + # change the default scheduler to single-threaded. This should not + # be particularly problematic, as only tiny test cases are run on Travis. + import dask + + dask.config.set( + scheduler="single-threaded" + ) # overwrite default with threaded scheduler + + +@contextlib.contextmanager +def logtime(tag, tag2=""): + logger.info(f"begin {tag} {tag2}") + t0 = time.time() + try: + yield + except Exception: + logger.error(f"error in {tag} after {timedelta(seconds=time.time()-t0)} {tag2}") + raise + else: + logger.info(f"completed {tag} in {timedelta(seconds=time.time()-t0)} {tag2}") + + +class TimeLogger: + + aggregate_timing = {} + + def __init__(self, tag1): + self._time_point = self._time_start = time.time() + self._time_log = [] + self._tag1 = tag1 + + def mark(self, tag, ping=True, logger=None, suffix=""): + if ping: + now = time.time() + elapsed = now - self._time_point + self._time_log.append((tag, timedelta(seconds=elapsed))) + self._time_point = now + if logger is not None: + logger.info( + "elapsed time {0} {1} {2}".format( + tag, + timedelta(seconds=elapsed), + suffix, + ) + ) + else: + self._time_log.append((tag, "skipped")) + elapsed = 0 + if self._tag1: + tag = f"{self._tag1}.{tag}" + if tag not in self.aggregate_timing: + self.aggregate_timing[tag] = elapsed + else: + self.aggregate_timing[tag] += elapsed + + def summary(self, logger, tag, level=20, suffix=None): + gross_elaspsed = time.time() - self._time_start + if suffix: + msg = f"{tag} in {timedelta(seconds=gross_elaspsed)}: ({suffix})\n" + else: + msg = f"{tag} in {timedelta(seconds=gross_elaspsed)}: \n" + msgs = [] + for i in self._time_log: + j = timedelta(seconds=self.aggregate_timing[f"{self._tag1}.{i[0]}"]) + msgs.append(" - {0:24s} {1} [{2}]".format(*i, j)) + msg += "\n".join(msgs) + logger.log(level=level, msg=msg) + + @classmethod + def aggregate_summary( + cls, logger, heading="Aggregate Flow Timing Summary", level=20 + ): + msg = f"{heading}\n" + msgs = [] + for tag, elapsed in cls.aggregate_timing.items(): + msgs.append(" - {0:48s} {1}".format(tag, timedelta(seconds=elapsed))) + msg += "\n".join(msgs) + logger.log(level=level, msg=msg) + + +def only_simple(x, exclude_keys=()): + """ + All the values in a dict that are plain numbers, strings, or lists or tuples thereof. + """ + y = {} + for k, v in x.items(): + if k not in exclude_keys: + if isinstance(v, (Number, str)): + y[k] = v + elif isinstance(v, (list, tuple)): + if all(isinstance(j, (Number, str)) for j in v): + y[k] = v + return y + + +def get_flow( + spec, local_d, trace_label=None, choosers=None, interacts=None, zone_layer=None +): + extra_vars = only_simple(local_d) + orig_col_name = local_d.get("orig_col_name", None) + dest_col_name = local_d.get("dest_col_name", None) + stop_col_name = None + parking_col_name = None + timeframe = local_d.get("timeframe", "tour") + if timeframe == "trip": + orig_col_name = local_d.get("ORIGIN", orig_col_name) + dest_col_name = local_d.get("DESTINATION", dest_col_name) + parking_col_name = local_d.get("PARKING", parking_col_name) + if orig_col_name is None and "od_skims" in local_d: + orig_col_name = local_d["od_skims"].orig_key + if dest_col_name is None and "od_skims" in local_d: + dest_col_name = local_d["od_skims"].dest_key + if stop_col_name is None and "dp_skims" in local_d: + stop_col_name = local_d["dp_skims"].dest_key + local_d = size_terms_on_flow(local_d) + size_term_mapping = local_d.get("size_array", {}) + if "tt" in local_d: + aux_vars = local_d["tt"].export_for_numba() + else: + aux_vars = {} + flow = new_flow( + spec, + extra_vars, + orig_col_name, + dest_col_name, + trace_label, + timeframe=timeframe, + choosers=choosers, + stop_col_name=stop_col_name, + parking_col_name=parking_col_name, + size_term_mapping=size_term_mapping, + interacts=interacts, + zone_layer=zone_layer, + aux_vars=aux_vars, + ) + flow.tree.aux_vars = aux_vars + return flow + + +def should_invalidate_cache_file(cache_filename, *source_filenames): + """ + Check if a cache file should be invalidated. + + It should be invalidated if any source file has a modification time + more recent than the cache file modification time. + + Parameters + ---------- + cache_filename : Path-like + source_filenames : Collection[Path-like] + + Returns + ------- + bool + """ + try: + stat0 = os.stat(cache_filename) + except FileNotFoundError: + # cache file does not even exist + return True + for i in source_filenames: + stat1 = os.stat(i) + if stat0[ST_MTIME] < stat1[ST_MTIME]: + return True + return False + + +def scan_for_unused_names(tokens): + """ + Scan all spec files to find unused skim variable names. + + Parameters + ---------- + tokens : Collection[str] + + Returns + ------- + Set[str] + """ + configs_dir_list = inject.get_injectable("configs_dir") + configs_dir_list = ( + [configs_dir_list] if isinstance(configs_dir_list, str) else configs_dir_list + ) + assert isinstance(configs_dir_list, list) + + for directory in configs_dir_list: + logger.debug(f"scanning for unused skims in {directory}") + filenames = glob.glob(os.path.join(directory, "*.csv")) + for filename in filenames: + with open(filename, "rt") as f: + content = f.read() + missing_tokens = set() + for t in tokens: + if t not in content: + missing_tokens.add(t) + tokens = missing_tokens + if not tokens: + return tokens + return tokens + + +@inject.injectable(cache=True) +def skim_dataset_dict(skim_dataset): + from .skim_dataset import SkimDataset + + return SkimDataset(skim_dataset) + + +def skims_mapping( + orig_col_name, + dest_col_name, + timeframe="tour", + stop_col_name=None, + parking_col_name=None, + zone_layer=None, +): + logger.info("loading skims_mapping") + logger.info(f"- orig_col_name: {orig_col_name}") + logger.info(f"- dest_col_name: {dest_col_name}") + logger.info(f"- stop_col_name: {stop_col_name}") + skim_dataset = inject.get_injectable("skim_dataset") + if zone_layer == "maz" or zone_layer is None: + odim = "omaz" if "omaz" in skim_dataset.dims else "otaz" + ddim = "dmaz" if "dmaz" in skim_dataset.dims else "dtaz" + elif zone_layer == "taz": + odim = "otaz" + ddim = "dtaz" + if "omaz" in skim_dataset.dims: + # strip out all MAZ-specific features of the skim_dataset + dropdims = ["omaz", "dmaz"] + skim_dataset = skim_dataset.drop_dims(dropdims, errors="ignore") + for dd in dropdims: + if f"dim_redirection_{dd}" in skim_dataset.attrs: + del skim_dataset.attrs[f"dim_redirection_{dd}"] + for attr_name in list(skim_dataset.attrs): + if attr_name.startswith("blend"): + del skim_dataset.attrs[attr_name] + + else: + raise ValueError(f"unknown zone layer {zone_layer!r}") + if zone_layer: + logger.info(f"- zone_layer: {zone_layer}") + if ( + orig_col_name is not None + and dest_col_name is not None + and stop_col_name is None + and parking_col_name is None + ): + if timeframe == "timeless": + return dict( + skims=skim_dataset, + relationships=( + f"df._orig_col_name -> skims.{odim}", + f"df._dest_col_name -> skims.{ddim}", + ), + ) + if timeframe == "timeless_directional": + return dict( + od_skims=skim_dataset, + do_skims=skim_dataset, + relationships=( + f"df._orig_col_name -> od_skims.{odim}", + f"df._dest_col_name -> od_skims.{ddim}", + f"df._dest_col_name -> do_skims.{odim}", + f"df._orig_col_name -> do_skims.{ddim}", + ), + ) + elif timeframe == "trip": + return dict( + odt_skims=skim_dataset, + dot_skims=skim_dataset, + od_skims=skim_dataset, + relationships=( + f"df._orig_col_name -> odt_skims.{odim}", + f"df._dest_col_name -> odt_skims.{ddim}", + "df.trip_period -> odt_skims.time_period", + f"df._dest_col_name -> dot_skims.{odim}", + f"df._orig_col_name -> dot_skims.{ddim}", + "df.trip_period -> dot_skims.time_period", + f"df._orig_col_name -> od_skims.{odim}", + f"df._dest_col_name -> od_skims.{ddim}", + ), + ) + else: + return dict( + # TODO:SHARROW: organize dimensions. + odt_skims=skim_dataset, + dot_skims=skim_dataset, + odr_skims=skim_dataset, + dor_skims=skim_dataset, + od_skims=skim_dataset, + relationships=( + f"df._orig_col_name -> odt_skims.{odim}", + f"df._dest_col_name -> odt_skims.{ddim}", + "df.out_period @ odt_skims.time_period", + f"df._dest_col_name -> dot_skims.{odim}", + f"df._orig_col_name -> dot_skims.{ddim}", + "df.in_period @ dot_skims.time_period", + f"df._orig_col_name -> odr_skims.{odim}", + f"df._dest_col_name -> odr_skims.{ddim}", + "df.in_period @ odr_skims.time_period", + f"df._dest_col_name -> dor_skims.{odim}", + f"df._orig_col_name -> dor_skims.{ddim}", + "df.out_period @ dor_skims.time_period", + f"df._orig_col_name -> od_skims.{odim}", + f"df._dest_col_name -> od_skims.{ddim}", + ), + ) + elif stop_col_name is not None: # trip_destination + return dict( + od_skims=skim_dataset, + dp_skims=skim_dataset, + odt_skims=skim_dataset, + dot_skims=skim_dataset, + dpt_skims=skim_dataset, + pdt_skims=skim_dataset, + relationships=( + f"df._orig_col_name -> od_skims.{odim}", + f"df._dest_col_name -> od_skims.{ddim}", + f"df._dest_col_name -> dp_skims.{odim}", + f"df._stop_col_name -> dp_skims.{ddim}", + f"df._orig_col_name -> odt_skims.{odim}", + f"df._dest_col_name -> odt_skims.{ddim}", + "df.trip_period -> odt_skims.time_period", + f"df._dest_col_name -> dot_skims.{odim}", + f"df._orig_col_name -> dot_skims.{ddim}", + "df.trip_period -> dot_skims.time_period", + f"df._dest_col_name -> dpt_skims.{odim}", + f"df._stop_col_name -> dpt_skims.{ddim}", + "df.trip_period -> dpt_skims.time_period", + f"df._stop_col_name -> pdt_skims.{odim}", + f"df._dest_col_name -> pdt_skims.{ddim}", + "df.trip_period -> pdt_skims.time_period", + ), + ) + elif parking_col_name is not None: # parking location + return dict( + od_skims=skim_dataset, + do_skims=skim_dataset, + op_skims=skim_dataset, + pd_skims=skim_dataset, + odt_skims=skim_dataset, + dot_skims=skim_dataset, + opt_skims=skim_dataset, + pdt_skims=skim_dataset, + relationships=( + f"df._orig_col_name -> od_skims.{odim}", + f"df._dest_col_name -> od_skims.{ddim}", + f"df._dest_col_name -> do_skims.{odim}", + f"df._orig_col_name -> do_skims.{ddim}", + f"df._orig_col_name -> op_skims.{odim}", + f"df._park_col_name -> op_skims.{ddim}", + f"df._park_col_name -> pd_skims.{odim}", + f"df._dest_col_name -> pd_skims.{ddim}", + f"df._orig_col_name -> odt_skims.{odim}", + f"df._dest_col_name -> odt_skims.{ddim}", + "df.trip_period -> odt_skims.time_period", + f"df._dest_col_name -> dot_skims.{odim}", + f"df._orig_col_name -> dot_skims.{ddim}", + "df.trip_period -> dot_skims.time_period", + f"df._orig_col_name -> opt_skims.{odim}", + f"df._park_col_name -> opt_skims.{ddim}", + "df.trip_period -> opt_skims.time_period", + f"df._park_col_name -> pdt_skims.{odim}", + f"df._dest_col_name -> pdt_skims.{ddim}", + "df.trip_period -> pdt_skims.time_period", + ), + ) + else: + return {} + + +def new_flow( + spec, + extra_vars, + orig_col_name, + dest_col_name, + trace_label=None, + timeframe="tour", + choosers=None, + stop_col_name=None, + parking_col_name=None, + size_term_mapping=None, + interacts=None, + zone_layer=None, + aux_vars=None, +): + """ + Setup a new sharrow flow. + + Parameters + ---------- + spec : pandas.DataFrame + The spec, as usual for ActivitySim. The index should either be a basic + single-level index containing the expressions to be evaluated, or a + MultiIndex with at least "Expression" and "Label" levels. + extra_vars : Mapping + Extra values that are available to expressions and which are written + explicitly into compiled code (and cannot be changed later). + orig_col_name : str + The column from the choosers table that gives the origin zone index, + used to attach values from skims. + dest_col_name : str + The column from the choosers table that gives the destination zone index, + used to attach values from skims. + trace_label : str + A descriptive label + timeframe : {"tour", "timeless", "timeless_directional", "trip"}, default "tour" + A framework for how to treat the time and directionality of skims that + will be attached. + choosers : pandas.DataFrame + Attributes of the choosers, possibly interacted with attributes of the + alternatives. Generally this flow can and will be re-used by swapping + out the `choosers` for a new dataframe with the same columns and + different rows. + stop_col_name : str + The column from the choosers table that gives the stop zone index in + trip destination choice, used to attach values from skims. + parking_col_name : str + The column from the choosers table that gives the parking zone index, + used to attach values from skims. + size_term_mapping : Mapping + Size term arrays. + interacts : pd.DataFrame, optional + An unmerged interaction dataset, giving attributes of the alternatives + that are not conditional on the chooser. Use this when the choice model + has some variables that are conditional on the chooser (and included in + the `choosers` dataframe, and some variables that are conditional on the + alternative but not the chooser, and when every chooser has the same set + of possible alternatives. + zone_layer : {'taz', 'maz'}, default 'taz' + Specify which zone layer of the skims is to be used. You cannot use the + 'maz' zone layer in a one-zone model, but you can use the 'taz' layer in + a two- or three-zone model (e.g. for destination pre-sampling). + aux_vars : Mapping + Extra values that are available to expressions and which are written + only by reference into compiled code (and thus can be changed later). + + Returns + ------- + sharrow.Flow + """ + + with logtime(f"setting up flow {trace_label}"): + if choosers is None: + chooser_cols = [] + else: + chooser_cols = list(choosers.columns) + + cache_dir = os.path.join( + config.get_cache_dir(), + "__sharrowcache__", + ) + os.makedirs(cache_dir, exist_ok=True) + logger.debug(f"flow.cache_dir: {cache_dir}") + skims_mapping_ = skims_mapping( + orig_col_name, + dest_col_name, + timeframe, + stop_col_name, + parking_col_name=parking_col_name, + zone_layer=zone_layer, + ) + if size_term_mapping is None: + size_term_mapping = {} + + if interacts is None: + if choosers is None: + logger.info(f"empty flow on {trace_label}") + else: + logger.info(f"{len(choosers)} chooser rows on {trace_label}") + flow_tree = sh.DataTree(df=[] if choosers is None else choosers) + idx_name = choosers.index.name or "index" + rename_dataset_cols = { + idx_name: "chooserindex", + } + if orig_col_name is not None: + rename_dataset_cols[orig_col_name] = "_orig_col_name" + if dest_col_name is not None: + rename_dataset_cols[dest_col_name] = "_dest_col_name" + if stop_col_name is not None: + rename_dataset_cols[stop_col_name] = "_stop_col_name" + if parking_col_name is not None: + rename_dataset_cols[parking_col_name] = "_park_col_name" + + def _apply_filter(_dataset, renames: dict): + ds = _dataset.rename(renames).ensure_integer(renames.values()) + for _k, _v in renames.items(): + ds[_k] = ds[_v] + return ds + + from functools import partial + + flow_tree.replacement_filters[flow_tree.root_node_name] = partial( + _apply_filter, renames=rename_dataset_cols + ) + flow_tree.root_dataset = flow_tree.root_dataset # apply the filter + else: + logger.info( + f"{len(choosers)} chooser rows and {len(interacts)} interact rows on {trace_label}" + ) + top = sh.dataset.from_named_objects( + pd.RangeIndex(len(choosers), name="chooserindex"), + pd.RangeIndex(len(interacts), name="interactindex"), + ) + flow_tree = sh.DataTree(start=top) + rename_dataset_cols = { + orig_col_name: "_orig_col_name", + dest_col_name: "_dest_col_name", + } + if stop_col_name is not None: + rename_dataset_cols[stop_col_name] = "_stop_col_name" + if parking_col_name is not None: + rename_dataset_cols[parking_col_name] = "_park_col_name" + choosers_ = ( + sh.dataset.construct(choosers) + .rename_or_ignore(rename_dataset_cols) + .ensure_integer( + [ + "_orig_col_name", + "_dest_col_name", + "_stop_col_name", + "_park_col_name", + ] + ) + ) + for _k, _v in rename_dataset_cols.items(): + if _v in choosers_: + choosers_[_k] = choosers_[_v] + flow_tree.add_dataset( + "df", + choosers_, + f"start.chooserindex -> df.{next(iter(choosers_.dims))}", + ) + interacts_ = sh.dataset.construct(interacts).rename_or_ignore( + rename_dataset_cols + ) + flow_tree.add_dataset( + "interact_table", + interacts_, + f"start.interactindex -> interact_table.{next(iter(interacts_.dims))}", + ) + flow_tree.subspace_fallbacks["df"] = ["interact_table"] + + flow_tree.add_items(skims_mapping_) + flow_tree.add_items(size_term_mapping) + flow_tree.extra_vars = extra_vars + flow_tree.extra_funcs = ( + sharrow_tt_remaining_periods_available, + sharrow_tt_previous_tour_begins, + sharrow_tt_previous_tour_ends, + sharrow_tt_adjacent_window_after, + sharrow_tt_adjacent_window_before, + sharrow_tt_max_time_block_available, + ) + flow_tree.aux_vars = aux_vars + + # - eval spec expressions + if isinstance(spec.index, pd.MultiIndex): + # spec MultiIndex with expression and label + exprs = spec.index.get_level_values(SPEC_EXPRESSION_NAME) + labels = spec.index.get_level_values(SPEC_LABEL_NAME) + else: + exprs = spec.index + labels = exprs + + defs = {} + # duplicate labels cause problems for sharrow, so we need to dedupe + existing_labels = set() + for (expr, label) in zip(exprs, labels): + while label in existing_labels: + label = label + "_" + existing_labels.add(label) + if expr[0] == "@": + if label == expr: + if expr[1:].isidentifier(): + defs[expr[1:] + "_"] = expr[1:] + else: + defs[expr[1:]] = expr[1:] + else: + defs[label] = expr[1:] + elif expr[0] == "_" and "@" in expr: + # - allow temps of form _od_DIST@od_skim['DIST'] + target = expr[: expr.index("@")] + rhs = expr[expr.index("@") + 1 :] + defs[target] = rhs + else: + if label == expr and expr.isidentifier(): + defs[expr + "_"] = expr + else: + defs[label] = expr + + readme = f""" + activitysim version: {__version__} + trace label: {trace_label} + orig_col_name: {orig_col_name} + dest_col_name: {dest_col_name} + expressions:""" + for (expr, label) in zip(exprs, labels): + readme += f"\n - {label}: {expr}" + if extra_vars: + readme += "\n extra_vars:" + for i, v in extra_vars.items(): + readme += f"\n - {i}: {v}" + + logger.info(f"setting up sharrow flow {trace_label}") + extra_hash_data = () + if zone_layer: + extra_hash_data += (zone_layer,) + return flow_tree.setup_flow( + defs, + cache_dir=cache_dir, + readme=readme[1:], # remove leading newline + flow_library=_FLOWS, + extra_hash_data=extra_hash_data, + hashing_level=0, + boundscheck=False, + ) + + +def size_terms_on_flow(locals_d): + if "size_terms_array" in locals_d: + # skim_dataset = inject.get_injectable('skim_dataset') + dest_col_name = locals_d["od_skims"].dest_key + a = sh.Dataset( + { + "arry": sh.DataArray( + locals_d["size_terms_array"], + dims=["stoptaz", "purpose_index"], + coords={ + "stoptaz": np.arange( + locals_d["size_terms_array"].shape[0] + ), # TODO: this assumes zero-based array of choices, is this always right? + }, + ) + } + ) + # a = a.reindex(stoptaz=skim_dataset.coords['dtaz'].values) # TODO {ddim}? + locals_d["size_array"] = dict( + size_terms=a, + relationships=( + "df._dest_col_name -> size_terms.stoptaz", + "df.purpose_index_num -> size_terms.purpose_index", + ), + ) + return locals_d + + +def apply_flow( + spec, + choosers, + locals_d=None, + trace_label=None, + required=False, + interacts=None, + zone_layer=None, +): + if sh is None: + return None, None + if locals_d is None: + locals_d = {} + with logtime("apply_flow"): + try: + flow = get_flow( + spec, + locals_d, + trace_label, + choosers=choosers, + interacts=interacts, + zone_layer=zone_layer, + ) + except ValueError as err: + if "unable to rewrite" in str(err): + logger.error(f"error in apply_flow: {err!s}") + if required: + raise + return None, None + else: + raise + with logtime("flow.load", trace_label or ""): + try: + flow_result = flow.dot( + coefficients=spec.values.astype(np.float32), + dtype=np.float32, + compile_watch=True, + ) + # TODO: are there remaining internal arrays in dot that need to be + # passed out to be seen by the dynamic chunker before they are freed? + except ValueError as err: + if "could not convert" in str(err): + logger.error(f"error in apply_flow: {err!s}") + if required: + raise + return None, flow + raise + except Exception as err: + logger.error(f"error in apply_flow: {err!s}") + # index_keys = self.shared_data.meta_match_names_idx.keys() + # logger.debug(f"Flow._get_indexes: {index_keys}") + raise + if flow.compiled_recently: + tracing.timing_notes.add(f"compiled:{flow.name}") + return flow_result, flow diff --git a/activitysim/core/inject.py b/activitysim/core/inject.py index 6e2e9d5fa1..208a5658fd 100644 --- a/activitysim/core/inject.py +++ b/activitysim/core/inject.py @@ -1,6 +1,7 @@ # ActivitySim # See full license in LICENSE.txt. import logging +import warnings from orca import orca @@ -8,6 +9,7 @@ _DECORATED_TABLES = {} _DECORATED_COLUMNS = {} _DECORATED_INJECTABLES = {} +_BROADCASTS = [] # we want to allow None (any anyting else) as a default value, so just choose an improbable string @@ -25,6 +27,31 @@ def decorator(func): assert not _DECORATED_STEPS.get(name, False), ( "step '%s' already decorated." % name ) + if _DECORATED_STEPS.get(name, False): + warnings.warn( + f"step {name!r} already exists, ignoring default implementation." + ) + else: + _DECORATED_STEPS[name] = func + orca.add_step(name, func) + + return func + + return decorator + + +def custom_step(): + """ + This decorator allows custom steps to potentially overload existing steps. + """ + + def decorator(func): + name = func.__name__ + + logger.debug("inject step %s" % name) + + if _DECORATED_STEPS.get(name, False): + warnings.warn(f"step {name!r} already exists, overwriting it.") _DECORATED_STEPS[name] = func orca.add_step(name, func) @@ -109,6 +136,17 @@ def add_injectable(name, injectable, cache=False): def broadcast( cast, onto, cast_on=None, onto_on=None, cast_index=False, onto_index=False ): + _BROADCASTS.append( + [ + (cast, onto), + dict( + cast_on=cast_on, + onto_on=onto_on, + cast_index=cast_index, + onto_index=onto_index, + ), + ] + ) return orca.broadcast( cast, onto, @@ -145,9 +183,12 @@ def remove_injectable(name): orca._INJECTABLES.pop(name, None) -def reinject_decorated_tables(): +def reinject_decorated_tables(steps=False): """ reinject the decorated tables (and columns) + + This function can be used to completely reset the global state for + ActivitySim. """ logger.info("reinject_decorated_tables") @@ -157,6 +198,9 @@ def reinject_decorated_tables(): orca._COLUMNS.clear() orca._TABLE_CACHE.clear() orca._COLUMN_CACHE.clear() + if steps: + orca._STEPS.clear() + orca._BROADCASTS.clear() for name, func in _DECORATED_TABLES.items(): logger.debug("reinject decorated table %s" % name) @@ -171,6 +215,13 @@ def reinject_decorated_tables(): logger.debug("reinject decorated injectable %s" % name) orca.add_injectable(name, args["func"], cache=args["cache"]) + if steps: + for name, func in _DECORATED_STEPS.items(): + logger.debug("reinject decorated step %s" % name) + orca.add_step(name, func) + for arg, kwarg in _BROADCASTS: + orca.broadcast(*arg, **kwarg) + def clear_cache(): return orca.clear_cache() diff --git a/activitysim/core/input.py b/activitysim/core/input.py index 73183f3e91..f0d6a9c028 100644 --- a/activitysim/core/input.py +++ b/activitysim/core/input.py @@ -85,6 +85,7 @@ def read_from_table_info(table_info): column_map = table_info.get("column_map", None) keep_columns = table_info.get("keep_columns", None) rename_columns = table_info.get("rename_columns", None) + recode_columns = table_info.get("recode_columns", None) csv_dtypes = table_info.get("dtypes", {}) # don't require a redundant index_col directive for canonical tables @@ -156,6 +157,31 @@ def read_from_table_info(table_info): logger.debug("renaming columns: %s" % rename_columns) df.rename(columns=rename_columns, inplace=True) + # recode columns, can simplify data structure + if recode_columns and config.setting("recode_pipeline_columns", True): + for colname, recode_instruction in recode_columns.items(): + logger.info(f"recoding column {colname}: {recode_instruction}") + if recode_instruction == "zero-based": + remapper = {j: i for i, j in enumerate(sorted(set(df[colname])))} + df[f"_original_{colname}"] = df[colname] + df[colname] = df[colname].apply(remapper.get) + if keep_columns: + keep_columns.append(f"_original_{colname}") + if tablename == "land_use" and colname == canonical_index_col: + # We need to keep track if we have recoded the land_use + # table's index to zero-based, as we need to disable offset + # processing for legacy skim access. + config.override_setting("offset_preprocessing", True) + else: + source_table, lookup_col = recode_instruction.split(".") + parent_table = inject.get_table(source_table) + try: + map_col = parent_table[f"_original_{lookup_col}"] + except KeyError: + map_col = parent_table[lookup_col] + remapper = dict(zip(map_col, parent_table.index)) + df[colname] = df[colname].apply(remapper.get) + # set index if index_col is not None: if index_col in df.columns: diff --git a/activitysim/core/interaction_sample.py b/activitysim/core/interaction_sample.py index 6d5d87edf3..a385b94718 100644 --- a/activitysim/core/interaction_sample.py +++ b/activitysim/core/interaction_sample.py @@ -1,13 +1,11 @@ # ActivitySim # See full license in LICENSE.txt. import logging -from builtins import range -from math import ceil import numpy as np import pandas as pd -from . import chunk, interaction_simulate, logit, pipeline, tracing +from . import chunk, config, interaction_simulate, logit, pipeline, tracing from .simulate import set_skim_wrapper_targets logger = logging.getLogger(__name__) @@ -62,13 +60,13 @@ def make_sample_choices( probs = probs[~zero_probs] choosers = choosers[~zero_probs] - cum_probs_array = probs.values.cumsum(axis=1) - chunk.log_df(trace_label, "cum_probs_array", cum_probs_array) - - # alt probs in convenient layout to return prob of chose alternative - # (same layout as cum_probs_arr) - alt_probs_array = probs.values.flatten() - chunk.log_df(trace_label, "alt_probs_array", alt_probs_array) + # cum_probs_array = probs.values.cumsum(axis=1) + # chunk.log_df(trace_label, 'cum_probs_array', cum_probs_array) + # + # # alt probs in convenient layout to return prob of chose alternative + # # (same layout as cum_probs_arr) + # alt_probs_array = probs.values.flatten() + # chunk.log_df(trace_label, 'alt_probs_array', alt_probs_array) # get sample_size rands for each chooser rands = pipeline.get_rn_generator().random_for_df(probs, n=sample_size) @@ -76,69 +74,81 @@ def make_sample_choices( # transform as we iterate over alternatives # reshape so rands[i] is in broadcastable (2-D) shape for cum_probs_arr # i.e rands[i] is a 2-D array of one alt choice rand for each chooser - rands = rands.T.reshape(sample_size, -1, 1) + # rands = rands.T #.reshape(sample_size, -1, 1) chunk.log_df(trace_label, "rands", rands) - # the alternative value chosen - # WHY SHOULD CHOICES COL HAVE TO BE TYPE INT??? + # + # # the alternative value chosen # choices_array = np.empty([sample_size, len(choosers)]).astype(int) - choices_array = np.empty([sample_size, len(choosers)]).astype( - alternatives.index.dtype + # # chunk log these later after we populate them... + # + # # the probability of the chosen alternative + # choice_probs_array = np.empty([sample_size, len(choosers)]) + # # chunk log these later after we populate them... + # + # alts = np.tile(alternatives.index.values, len(choosers)) + # chunk.log_df(trace_label, 'alts', alts) + # + # # FIXME - do this all at once rather than iterate? + # for i in range(sample_size): + # + # # FIXME - do this in numpy, not pandas? + # + # # rands for this alt in broadcastable shape + # r = rands[i] + # + # # position of first occurrence of positive value + # positions = np.argmax(cum_probs_array > r, axis=1) + # + # # FIXME - leave positions as numpy array, not pandas series? + # # positions is series with the chosen alternative represented as a column index in probs + # # which is an integer between zero and num alternatives in the alternative sample + # positions = pd.Series(positions, index=probs.index) + # + # # need to get from an integer offset into the alternative sample to the alternative index + # # that is, we want the index value of the row that is offset by rows into the + # # tranche of this choosers alternatives created by cross join of alternatives and choosers + # + # # offsets is the offset into model_design df of first row of chooser alternatives + # offsets = np.arange(len(positions)) * alternative_count + # + # # choices and choice_probs have one element per chooser and is in same order as choosers + # choices_array[i] = np.take(alts, positions + offsets) + # choice_probs_array[i] = np.take(alt_probs_array, positions + offsets) + # + # del positions + # del offsets + # + # chunk.log_df(trace_label, 'choices_array', choices_array) + # chunk.log_df(trace_label, 'choice_probs_array', choice_probs_array) + # + # del alts + # chunk.log_df(trace_label, 'alts', None) + # del cum_probs_array + # chunk.log_df(trace_label, 'cum_probs_array', None) + # del alt_probs_array + # chunk.log_df(trace_label, 'alt_probs_array', None) + + # TODO: is `sample_choices_maker` more efficient? The order of samples changes, might change repro-randoms + from .choosing import sample_choices_maker_preserve_ordering + + choices_array, choice_probs_array = sample_choices_maker_preserve_ordering( + probs.values, + rands, + alternatives.index.values, ) - # chunk log these later after we populate them... - - # the probability of the chosen alternative - choice_probs_array = np.empty([sample_size, len(choosers)]) - # chunk log these later after we populate them... - - alts = np.tile(alternatives.index.values, len(choosers)) - chunk.log_df(trace_label, "alts", alts) - - # FIXME - do this all at once rather than iterate? - for i in range(sample_size): - - # FIXME - do this in numpy, not pandas? - - # rands for this alt in broadcastable shape - r = rands[i] - - # position of first occurrence of positive value - positions = np.argmax(cum_probs_array > r, axis=1) - - # FIXME - leave positions as numpy array, not pandas series? - # positions is series with the chosen alternative represented as a column index in probs - # which is an integer between zero and num alternatives in the alternative sample - positions = pd.Series(positions, index=probs.index) - - # need to get from an integer offset into the alternative sample to the alternative index - # that is, we want the index value of the row that is offset by rows into the - # tranche of this choosers alternatives created by cross join of alternatives and choosers - - # offsets is the offset into model_design df of first row of chooser alternatives - offsets = np.arange(len(positions)) * alternative_count - - # choices and choice_probs have one element per chooser and is in same order as choosers - choices_array[i] = np.take(alts, positions + offsets) - choice_probs_array[i] = np.take(alt_probs_array, positions + offsets) - - del positions - del offsets chunk.log_df(trace_label, "choices_array", choices_array) chunk.log_df(trace_label, "choice_probs_array", choice_probs_array) - del alts - chunk.log_df(trace_label, "alts", None) - del cum_probs_array - chunk.log_df(trace_label, "cum_probs_array", None) - del alt_probs_array - chunk.log_df(trace_label, "alt_probs_array", None) + # np.testing.assert_array_equal(choices_array, choices_array__) + # np.testing.assert_array_almost_equal(choice_probs_array, choice_probs_array__) # explode to one row per chooser.index, alt_zone_id choices_df = pd.DataFrame( { alt_col_name: choices_array.flatten(order="F"), - "rand": rands.flatten(order="F"), + "rand": rands.T.flatten(order="F"), "prob": choice_probs_array.flatten(order="F"), choosers.index.name: np.repeat(np.asanyarray(choosers.index), sample_size), } @@ -170,6 +180,7 @@ def _interaction_sample( skims=None, locals_d=None, trace_label=None, + zone_layer=None, ): """ Run a MNL simulation in the situation in which alternatives must @@ -227,6 +238,8 @@ def _interaction_sample( """ have_trace_targets = tracing.has_trace_targets(choosers) + trace_ids = None + trace_rows = None num_choosers = len(choosers.index) assert num_choosers > 0 @@ -251,76 +264,192 @@ def _interaction_sample( chooser_index_id = interaction_simulate.ALT_CHOOSER_ID if log_alt_losers else None + sharrow_enabled = config.setting("sharrow", False) + # - cross join choosers and alternatives (cartesian product) # for every chooser, there will be a row for each alternative # index values (non-unique) are from alternatives df alternative_count = alternatives.shape[0] - interaction_df = logit.interaction_dataset( - choosers, - alternatives, - sample_size=alternative_count, - chooser_index_id=chooser_index_id, - ) - chunk.log_df(trace_label, "interaction_df", interaction_df) - - assert alternative_count == len(interaction_df.index) / len(choosers.index) + interaction_utilities = None + interaction_utilities_sh = None + if sharrow_enabled: + + # # TRACE ONLY + # from . import inject + # traceable_table_ids = inject.get_injectable('traceable_table_ids', {}) + # + # if choosers.index.name == 'person_id' and 'persons' in traceable_table_ids: + # slicer_column_name = choosers.index.name + # targets = traceable_table_ids['persons'] + # elif 'household_id' in choosers.columns and 'households' in traceable_table_ids: + # slicer_column_name = 'household_id' + # targets = traceable_table_ids['households'] + # elif 'person_id' in choosers.columns and 'persons' in traceable_table_ids: + # slicer_column_name = 'person_id' + # targets = traceable_table_ids['persons'] + # else: + # print(choosers.columns) + # raise RuntimeError("interaction_trace_rows don't know how to slice index '%s'" + # % choosers.index.name) + # + # if slicer_column_name == choosers.index.name: + # trace_rows = np.in1d(choosers.index, targets) + # trace_ids = np.asanyarray(choosers[trace_rows].index) + # elif slicer_column_name == 'person_id': + # trace_rows = np.in1d(choosers['person_id'], targets) + # trace_ids = np.asanyarray(choosers[trace_rows].person_id) + # elif slicer_column_name == 'household_id': + # trace_rows = np.in1d(choosers['household_id'], targets) + # trace_ids = np.asanyarray(choosers[trace_rows].household_id) + # else: + # assert False + # + # trace_ids = (slicer_column_name, trace_ids) + # # /TRACE ONLY + + ( + interaction_utilities, + trace_eval_results, + ) = interaction_simulate.eval_interaction_utilities( + spec, + choosers, + locals_d, + trace_label, + trace_rows, + estimator=None, + log_alt_losers=log_alt_losers, + extra_data=alternatives, + zone_layer=zone_layer, + ) + chunk.log_df(trace_label, "interaction_utilities", interaction_utilities) + if sharrow_enabled == "test" or True: + interaction_utilities_sh, trace_eval_results_sh = ( + interaction_utilities, + trace_eval_results, + ) + if not sharrow_enabled or (sharrow_enabled == "test"): + interaction_df = logit.interaction_dataset( + choosers, + alternatives, + sample_size=alternative_count, + chooser_index_id=chooser_index_id, + ) - if skims is not None: - set_skim_wrapper_targets(interaction_df, skims) + chunk.log_df(trace_label, "interaction_df", interaction_df) - # evaluate expressions from the spec multiply by coefficients and sum - # spec is df with one row per spec expression and one col with utility coefficient - # column names of interaction_df match spec index values - # utilities has utility value for element in the cross product of choosers and alternatives - # interaction_utilities is a df with one utility column and one row per row in interaction_df - if have_trace_targets: - trace_rows, trace_ids = tracing.interaction_trace_rows( - interaction_df, choosers, alternative_count - ) + assert alternative_count == len(interaction_df.index) / len(choosers.index) - tracing.trace_df( - interaction_df[trace_rows], - tracing.extend_trace_label(trace_label, "interaction_df"), - slicer="NONE", - transpose=False, - ) - else: - trace_rows = trace_ids = None - - # interaction_utilities is a df with one utility column and one row per interaction_df row - ( - interaction_utilities, - trace_eval_results, - ) = interaction_simulate.eval_interaction_utilities( - spec, - interaction_df, - locals_d, - trace_label, - trace_rows, - estimator=None, - log_alt_losers=log_alt_losers, - ) - chunk.log_df(trace_label, "interaction_utilities", interaction_utilities) + if skims is not None: + set_skim_wrapper_targets(interaction_df, skims) - # ########### HWM - high water mark (point of max observed memory usage) + # evaluate expressions from the spec multiply by coefficients and sum + # spec is df with one row per spec expression and one col with utility coefficient + # column names of interaction_df match spec index values + # utilities has utility value for element in the cross product of choosers and alternatives + # interaction_utilities is a df with one utility column and one row per row in interaction_df + if have_trace_targets: + trace_rows, trace_ids = tracing.interaction_trace_rows( + interaction_df, choosers, alternative_count + ) - del interaction_df - chunk.log_df(trace_label, "interaction_df", None) + tracing.trace_df( + interaction_df[trace_rows], + tracing.extend_trace_label(trace_label, "interaction_df"), + slicer="NONE", + transpose=False, + ) + else: + trace_rows = trace_ids = None - if have_trace_targets: + # interaction_utilities is a df with one utility column and one row per interaction_df row + ( + interaction_utilities, + trace_eval_results, + ) = interaction_simulate.eval_interaction_utilities( + spec, + interaction_df, + locals_d, + trace_label, + trace_rows, + estimator=None, + log_alt_losers=log_alt_losers, + zone_layer=zone_layer, + ) + chunk.log_df(trace_label, "interaction_utilities", interaction_utilities) + + # ########### HWM - high water mark (point of max observed memory usage) + + del interaction_df + chunk.log_df(trace_label, "interaction_df", None) + + if sharrow_enabled == "test": + try: + if interaction_utilities_sh is not None: + np.testing.assert_allclose( + interaction_utilities_sh.values.reshape( + interaction_utilities.values.shape + ), + interaction_utilities.values, + rtol=1e-2, + atol=0, + err_msg="utility not aligned", + verbose=True, + ) + except AssertionError as err: + print(err) + misses = np.where( + ~np.isclose( + interaction_utilities_sh.values, + interaction_utilities.values, + rtol=1e-2, + atol=0, + ) + ) + _sh_util_miss1 = interaction_utilities_sh.values[ + tuple(m[0] for m in misses) + ] + _u_miss1 = interaction_utilities.values[tuple(m[0] for m in misses)] + diff = _sh_util_miss1 - _u_miss1 + if len(misses[0]) > interaction_utilities_sh.values.size * 0.01: + print("big problem") + print(misses) + if "nan location mismatch" in str(err): + print("nan location mismatch interaction_utilities_sh") + print(np.where(np.isnan(interaction_utilities_sh.values))) + print("nan location mismatch interaction_utilities legacy") + print(np.where(np.isnan(interaction_utilities.values))) + print("misses =>", misses) + j = 0 + while j < len(misses[0]): + print( + f"miss {j} {tuple(m[j] for m in misses)}:", + interaction_utilities_sh.values[tuple(m[j] for m in misses)], + "!=", + interaction_utilities.values[tuple(m[j] for m in misses)], + ) + j += 1 + if j > 10: + break + raise + + if have_trace_targets and trace_ids is not None: tracing.trace_interaction_eval_results( trace_eval_results, trace_ids, tracing.extend_trace_label(trace_label, "eval"), ) - tracing.trace_df( - interaction_utilities[trace_rows], - tracing.extend_trace_label(trace_label, "interaction_utilities"), - slicer="NONE", - transpose=False, - ) + if have_trace_targets and trace_rows is not None: + try: + tracing.trace_df( + interaction_utilities[trace_rows], + tracing.extend_trace_label(trace_label, "interaction_utilities"), + slicer="NONE", + transpose=False, + ) + except ValueError: + pass tracing.dump_df(DUMP, interaction_utilities, trace_label, "interaction_utilities") @@ -457,6 +586,7 @@ def interaction_sample( chunk_size=0, chunk_tag=None, trace_label=None, + zone_layer=None, ): """ @@ -542,6 +672,7 @@ def interaction_sample( skims=skims, locals_d=locals_d, trace_label=chunk_trace_label, + zone_layer=zone_layer, ) if choices.shape[0] > 0: diff --git a/activitysim/core/interaction_sample_simulate.py b/activitysim/core/interaction_sample_simulate.py index 776df91d47..92cf644122 100644 --- a/activitysim/core/interaction_sample_simulate.py +++ b/activitysim/core/interaction_sample_simulate.py @@ -121,7 +121,9 @@ def _interaction_sample_simulate( # here, alternatives is sparsely repeated once for each (non-dup) sample # we expect alternatives to have same index of choosers (but with duplicate index values) # so we just need to left join alternatives with choosers - assert alternatives.index.name == choosers.index.name + + # assert alternatives.index.name == choosers.index.name + # asserting the index names are the same tells us nothing about the underlying data so why? interaction_df = alternatives.join(choosers, how="left", rsuffix="_chooser") @@ -263,6 +265,16 @@ def _interaction_sample_simulate( if allow_zero_probs: zero_probs = probs.sum(axis=1) == 0 if zero_probs.any(): + # Debug tracing in depth + # _z_interaction_df = alternatives.loc[zero_probs].join( + # choosers.loc[zero_probs], how='left', rsuffix='_chooser' + # ) + # _z_trace_rows = np.ones(len(_z_interaction_df), dtype=np.bool) + # _z_interaction_utilities, _z_trace_eval_results \ + # = interaction_simulate.eval_interaction_utilities( + # spec, _z_interaction_df, locals_d, trace_label, + # _z_trace_rows, + # estimator=estimator, log_alt_losers=log_alt_losers) # FIXME this is kind of gnarly, but we force choice of first alt probs.loc[zero_probs, 0] = 1.0 @@ -294,7 +306,7 @@ def _interaction_sample_simulate( chunk.log_df(trace_label, "choices", choices) - if allow_zero_probs and zero_probs.any(): + if allow_zero_probs and zero_probs.any() and zero_prob_choice_val is not None: # FIXME this is kind of gnarly, patch choice for zero_probs choices.loc[zero_probs] = zero_prob_choice_val diff --git a/activitysim/core/interaction_simulate.py b/activitysim/core/interaction_simulate.py index 836fc4a348..49ad38a7e7 100644 --- a/activitysim/core/interaction_simulate.py +++ b/activitysim/core/interaction_simulate.py @@ -1,8 +1,10 @@ # ActivitySim # See full license in LICENSE.txt. import logging +import time from builtins import zip from collections import OrderedDict +from datetime import timedelta import numpy as np import pandas as pd @@ -17,7 +19,15 @@ def eval_interaction_utilities( - spec, df, locals_d, trace_label, trace_rows, estimator=None, log_alt_losers=False + spec, + df, + locals_d, + trace_label, + trace_rows, + estimator=None, + log_alt_losers=False, + extra_data=None, + zone_layer=None, ): """ Compute the utilities for a single-alternative spec evaluated in the context of df @@ -57,9 +67,42 @@ def eval_interaction_utilities( Will have the index of `df` and a single column of utilities """ + start_time = time.time() + trace_label = tracing.extend_trace_label(trace_label, "eval_interaction_utils") logger.info("Running eval_interaction_utilities on %s rows" % df.shape[0]) + # # extract expressions and labels from spec.index + # if isinstance(spec.index, pd.MultiIndex): + # exprs = spec.index.get_level_values(simulate.SPEC_EXPRESSION_NAME) + # labels = spec.index.get_level_values(simulate.SPEC_LABEL_NAME) + # else: + # exprs = spec.index + # labels = spec.index + # + # # rewrite tt.xxx from spec expression to use numba accelerated versions + # updated_exprs = [] + # for e in exprs: + # e = e.replace("tt.adjacent_window_before(", "tt_adjacent_window_before(tt, ") + # e = e.replace("tt.adjacent_window_after(", "tt_adjacent_window_after(tt, ") + # e = e.replace("tt.previous_tour_ends(", "tt_previous_tour_ends(tt, ") + # e = e.replace("tt.previous_tour_begins(", "tt_previous_tour_begins(tt, ") + # e = e.replace("tt.remaining_periods_available(", "tt_remaining_periods_available(tt, ") + # updated_exprs.append(e) + # exprs = updated_exprs + # + # from .flow import apply_flow + # from . import inject + # skim_dataset = inject.get_injectable('skim_dataset') + sharrow_enabled = config.setting("sharrow", False) + + # if trace_label.startswith("trip_destination"): + # sharrow_enabled = False + + logger.info(f"{trace_label} sharrow_enabled is {sharrow_enabled}") + + trace_eval_results = None + with chunk.chunk_log(trace_label): assert len(spec.columns) == 1 @@ -67,190 +110,491 @@ def eval_interaction_utilities( # avoid altering caller's passed-in locals_d parameter (they may be looping) locals_d = locals_d.copy() if locals_d is not None else {} - # add df for startswith('@') eval expressions - locals_d["df"] = df + # # add numba accelerated versions of timetable functions to locals_d + # locals_d.update( + # tt_adjacent_window_before=timetable.tt_adjacent_window_before, + # tt_adjacent_window_after=timetable.tt_adjacent_window_after, + # tt_previous_tour_ends=timetable.tt_previous_tour_ends, + # tt_previous_tour_begins=timetable.tt_previous_tour_begins, + # tt_remaining_periods_available=timetable.tt_remaining_periods_available, + # ) - def to_series(x): - if np.isscalar(x): - return pd.Series([x] * len(df), index=df.index) - if isinstance(x, np.ndarray): - return pd.Series(x, index=df.index) - return x + utilities = None - if trace_rows is not None and trace_rows.any(): - # # convert to numpy array so we can slice ndarrays as well as series - # trace_rows = np.asanyarray(trace_rows) - assert type(trace_rows) == np.ndarray - trace_eval_results = OrderedDict() - else: - trace_eval_results = None + from .flow import TimeLogger - check_for_variability = config.setting("check_for_variability") + timelogger = TimeLogger("interaction_simulate") - # need to be able to identify which variables causes an error, which keeps - # this from being expressed more parsimoniously + # add df for startswith('@') eval expressions + locals_d["df"] = df - utilities = pd.DataFrame({"utility": 0.0}, index=df.index) + if sharrow_enabled: - chunk.log_df(trace_label, "eval.utilities", utilities) + from .flow import apply_flow - no_variability = has_missing_vals = 0 + spec_sh = spec.copy() - if estimator: - # ensure alt_id from interaction_dataset is available in expression_values_df for - # estimator.write_interaction_expression_values and eventual omnibus table assembly - alt_id = estimator.get_alt_id() - assert alt_id in df.columns - expression_values_df = df[[alt_id]] + def replace_in_index_level(mi, level, *repls): + if isinstance(mi, pd.MultiIndex): + level = mi._get_level_number(level) + content = list(mi.levels[level]) + new_content = [] + for i in content: + for repl in repls: + i = i.replace(*repl) + new_content.append(i) + return mi.set_levels(new_content, level=level) + else: + new_content = [] + for i in mi: + for repl in repls: + i = i.replace(*repl) + new_content.append(i) + return new_content + + spec_sh.index = replace_in_index_level( + spec_sh.index, + simulate.SPEC_EXPRESSION_NAME, + ( + "tt.adjacent_window_before(", + "sharrow_tt_adjacent_window_before(tt_windows, tt_row_mapper, tt_col_mapper, ", + ), + ( + "tt.adjacent_window_after(", + "sharrow_tt_adjacent_window_after(tt_windows, tt_row_mapper, tt_col_mapper, ", + ), + ( + "tt.previous_tour_ends(", + "sharrow_tt_previous_tour_ends(tt_windows, tt_row_mapper, tt_col_mapper, ", + ), + ( + "tt.previous_tour_begins(", + "sharrow_tt_previous_tour_begins(tt_windows, tt_row_mapper, tt_col_mapper, ", + ), + ( + "tt.remaining_periods_available(", + "sharrow_tt_remaining_periods_available(tt_windows, tt_row_mapper, ", + ), + ( + "tt.max_time_block_available(", + "sharrow_tt_max_time_block_available(tt_windows, tt_row_mapper, ", + ), + ) - # FIXME estimation_requires_chooser_id_in_df_column - # estimation requires that chooser_id is either in index or a column of interaction_dataset - # so it can be reformatted (melted) and indexed by chooser_id and alt_id - # we assume caller has this under control if index is named - # bug - location choice has df index_name zone_id but should be person_id???? - if df.index.name is None: - chooser_id = estimator.get_chooser_id() - assert ( - chooser_id in df.columns - ), "Expected to find choose_id column '%s' in interaction dataset" % ( - chooser_id, + # need to zero out any coefficients on temp vars + if isinstance(spec_sh.index, pd.MultiIndex): + exprs = spec_sh.index.get_level_values(simulate.SPEC_EXPRESSION_NAME) + labels = spec_sh.index.get_level_values(simulate.SPEC_LABEL_NAME) + else: + exprs = spec_sh.index + labels = spec_sh.index + for n, (expr, label) in enumerate(zip(exprs, labels)): + if expr.startswith("_") and "@" in expr: + spec_sh.iloc[n, 0] = 0.0 + + for i1, i2 in zip(exprs, labels): + logger.debug(f" - expr: {i1}: {i2}") + + timelogger.mark("sharrow preamble", True, logger, trace_label) + + sh_util, sh_flow = apply_flow( + spec_sh, + df, + locals_d, + trace_label, + interacts=extra_data, + zone_layer=zone_layer, + ) + if sh_util is not None: + chunk.log_df(trace_label, "sh_util", sh_util) + utilities = pd.DataFrame( + {"utility": sh_util.reshape(-1)}, + index=df.index if extra_data is None else None, ) - assert df.index.name is None - expression_values_df[chooser_id] = df[chooser_id] + chunk.log_df(trace_label, "sh_util", None) # hand off to caller - if isinstance(spec.index, pd.MultiIndex): - exprs = spec.index.get_level_values(simulate.SPEC_EXPRESSION_NAME) - labels = spec.index.get_level_values(simulate.SPEC_LABEL_NAME) + timelogger.mark("sharrow flow", True, logger, trace_label) else: - exprs = spec.index - labels = spec.index - - for expr, label, coefficient in zip(exprs, labels, spec.iloc[:, 0]): - try: - - # - allow temps of form _od_DIST@od_skim['DIST'] - if expr.startswith("_"): + sh_util, sh_flow = None, None + timelogger.mark("sharrow flow", False) + + if ( + utilities is None + or estimator + or (sharrow_enabled == "test" and extra_data is None) + ): + + def to_series(x): + if np.isscalar(x): + return pd.Series([x] * len(df), index=df.index) + if isinstance(x, np.ndarray): + return pd.Series(x, index=df.index) + return x + + if trace_rows is not None and trace_rows.any(): + # # convert to numpy array so we can slice ndarrays as well as series + # trace_rows = np.asanyarray(trace_rows) + assert type(trace_rows) == np.ndarray + trace_eval_results = OrderedDict() + else: + trace_eval_results = None + + check_for_variability = config.setting("check_for_variability") + + # need to be able to identify which variables causes an error, which keeps + # this from being expressed more parsimoniously + + utilities = pd.DataFrame({"utility": 0.0}, index=df.index) + + chunk.log_df(trace_label, "eval.utilities", utilities) + + no_variability = has_missing_vals = 0 + + if estimator: + # ensure alt_id from interaction_dataset is available in expression_values_df for + # estimator.write_interaction_expression_values and eventual omnibus table assembly + alt_id = estimator.get_alt_id() + assert alt_id in df.columns + expression_values_df = df[[alt_id]] + + # FIXME estimation_requires_chooser_id_in_df_column + # estimation requires that chooser_id is either in index or a column of interaction_dataset + # so it can be reformatted (melted) and indexed by chooser_id and alt_id + # we assume caller has this under control if index is named + # bug - location choice has df index_name zone_id but should be person_id???? + if df.index.name is None: + chooser_id = estimator.get_chooser_id() + assert chooser_id in df.columns, ( + "Expected to find choose_id column '%s' in interaction dataset" + % (chooser_id,) + ) + assert df.index.name is None + expression_values_df[chooser_id] = df[chooser_id] + + if isinstance(spec.index, pd.MultiIndex): + exprs = spec.index.get_level_values(simulate.SPEC_EXPRESSION_NAME) + labels = spec.index.get_level_values(simulate.SPEC_LABEL_NAME) + else: + exprs = spec.index + labels = spec.index + + for expr, label, coefficient in zip(exprs, labels, spec.iloc[:, 0]): + try: + + # - allow temps of form _od_DIST@od_skim['DIST'] + if expr.startswith("_"): + + target = expr[: expr.index("@")] + rhs = expr[expr.index("@") + 1 :] + v = to_series(eval(rhs, globals(), locals_d)) + + # update locals to allows us to ref previously assigned targets + locals_d[target] = v + chunk.log_df( + trace_label, target, v + ) # track temps stored in locals + + if trace_eval_results is not None: + trace_eval_results[expr] = v[trace_rows] + + # don't add temps to utility sums + # they have a non-zero dummy coefficient to avoid being removed from spec as NOPs + continue + + if expr.startswith("@"): + v = to_series(eval(expr[1:], globals(), locals_d)) + else: + v = df.eval(expr, resolvers=[locals_d]) + + if check_for_variability and v.std() == 0: + logger.info( + "%s: no variability (%s) in: %s" + % (trace_label, v.iloc[0], expr) + ) + no_variability += 1 + + # FIXME - how likely is this to happen? Not sure it is really a problem? + if ( + check_for_variability + and np.count_nonzero(v.isnull().values) > 0 + ): + logger.info("%s: missing values in: %s" % (trace_label, expr)) + has_missing_vals += 1 + + if estimator: + # in case we modified expression_values_df index + expression_values_df.insert( + loc=len(expression_values_df.columns), + column=label, + value=v.values if isinstance(v, pd.Series) else v, + ) - target = expr[: expr.index("@")] - rhs = expr[expr.index("@") + 1 :] - v = to_series(eval(rhs, globals(), locals_d)) + utility = (v * coefficient).astype("float") - # update locals to allows us to ref previously assigned targets - locals_d[target] = v - chunk.log_df(trace_label, target, v) # track temps stored in locals + if log_alt_losers: - if trace_eval_results is not None: - trace_eval_results[expr] = v[trace_rows] + assert ALT_CHOOSER_ID in df + max_utils_by_chooser = utility.groupby(df[ALT_CHOOSER_ID]).max() - # don't add temps to utility sums - # they have a non-zero dummy coefficient to avoid being removed from spec as NOPs - continue + if (max_utils_by_chooser < simulate.ALT_LOSER_UTIL).any(): - if expr.startswith("@"): - v = to_series(eval(expr[1:], globals(), locals_d)) - else: - v = df.eval(expr) + losers = max_utils_by_chooser[ + max_utils_by_chooser < simulate.ALT_LOSER_UTIL + ] + logger.warning( + f"{trace_label} - {len(losers)} choosers of {len(max_utils_by_chooser)} " + f"with prohibitive utilities for all alternatives for expression: {expr}" + ) - if check_for_variability and v.std() == 0: - logger.info( - "%s: no variability (%s) in: %s" - % (trace_label, v.iloc[0], expr) - ) - no_variability += 1 - - # FIXME - how likely is this to happen? Not sure it is really a problem? - if check_for_variability and np.count_nonzero(v.isnull().values) > 0: - logger.info("%s: missing values in: %s" % (trace_label, expr)) - has_missing_vals += 1 - - if estimator: - # in case we modified expression_values_df index - expression_values_df.insert( - loc=len(expression_values_df.columns), - column=label, - value=v.values if isinstance(v, pd.Series) else v, - ) + # loser_df = df[df[ALT_CHOOSER_ID].isin(losers.index)] + # print(f"\nloser_df\n{loser_df}\n") + # print(f"\nloser_max_utils_by_chooser\n{losers}\n") + # bug - utility = (v * coefficient).astype("float") + del max_utils_by_chooser - if log_alt_losers: + utilities.utility.values[:] += utility - assert ALT_CHOOSER_ID in df - max_utils_by_chooser = utility.groupby(df[ALT_CHOOSER_ID]).max() + if trace_eval_results is not None: - if (max_utils_by_chooser < simulate.ALT_LOSER_UTIL).any(): + # expressions should have been uniquified when spec was read + # (though we could do it here if need be...) + # expr = assign.uniquify_key(trace_eval_results, expr, template="{} # ({})") + assert expr not in trace_eval_results - losers = max_utils_by_chooser[ - max_utils_by_chooser < simulate.ALT_LOSER_UTIL - ] - logger.warning( - f"{trace_label} - {len(losers)} choosers of {len(max_utils_by_chooser)} " - f"with prohibitive utilities for all alternatives for expression: {expr}" + trace_eval_results[expr] = v[trace_rows] + k = "partial utility (coefficient = %s) for %s" % ( + coefficient, + expr, ) + trace_eval_results[k] = v[trace_rows] * coefficient - # loser_df = df[df[ALT_CHOOSER_ID].isin(losers.index)] - # print(f"\nloser_df\n{loser_df}\n") - # print(f"\nloser_max_utils_by_chooser\n{losers}\n") - # bug + del v + # chunk.log_df(trace_label, 'v', None) - del max_utils_by_chooser + except Exception as err: + logger.exception( + f"{trace_label} - {type(err).__name__} ({str(err)}) evaluating: {str(expr)}" + ) + raise err - utilities.utility += utility + if estimator: + estimator.log( + "eval_interaction_utilities write_interaction_expression_values %s" + % trace_label + ) + estimator.write_interaction_expression_values(expression_values_df) + del expression_values_df - if trace_eval_results is not None: + if no_variability > 0: + logger.warning( + "%s: %s columns have no variability" % (trace_label, no_variability) + ) - # expressions should have been uniquified when spec was read - # (though we could do it here if need be...) - # expr = assign.uniquify_key(trace_eval_results, expr, template="{} # ({})") - assert expr not in trace_eval_results + if has_missing_vals > 0: + logger.warning( + "%s: %s columns have missing values" + % (trace_label, has_missing_vals) + ) - trace_eval_results[expr] = v[trace_rows] - k = "partial utility (coefficient = %s) for %s" % ( - coefficient, - expr, - ) - trace_eval_results[k] = v[trace_rows] * coefficient + if trace_eval_results is not None: + trace_eval_results["total utility"] = utilities.utility[trace_rows] - del v - # chunk.log_df(trace_label, 'v', None) + trace_eval_results = pd.DataFrame.from_dict(trace_eval_results) + trace_eval_results.index = df[trace_rows].index - except Exception as err: - logger.exception( - f"{trace_label} - {type(err).__name__} ({str(err)}) evaluating: {str(expr)}" + # add df columns to trace_results + trace_eval_results = pd.concat( + [df[trace_rows], trace_eval_results], axis=1 ) - raise err + chunk.log_df(trace_label, "eval.trace_eval_results", trace_eval_results) - if estimator: - estimator.log( - "eval_interaction_utilities write_interaction_expression_values %s" - % trace_label - ) - estimator.write_interaction_expression_values(expression_values_df) - del expression_values_df + chunk.log_df(trace_label, "v", None) + chunk.log_df(trace_label, "eval.utilities", None) # out of out hands... + chunk.log_df(trace_label, "eval.trace_eval_results", None) - if no_variability > 0: - logger.warning( - "%s: %s columns have no variability" % (trace_label, no_variability) - ) + timelogger.mark("regular interact flow", True, logger, trace_label) + else: + timelogger.mark("regular interact flow", False) - if has_missing_vals > 0: - logger.warning( - "%s: %s columns have missing values" % (trace_label, has_missing_vals) + # + # Sharrow tracing + # + if sh_flow is not None and trace_rows is not None and trace_rows.any(): + assert type(trace_rows) == np.ndarray + sh_utility_fat = sh_flow.load_dataarray( + # sh_flow.tree.replace_datasets( + # df=df.iloc[trace_rows], + # ), + dtype=np.float32, ) + sh_utility_fat = sh_utility_fat[trace_rows, :] + sh_utility_fat = sh_utility_fat.to_dataframe("vals") + try: + sh_utility_fat = sh_utility_fat.unstack("expressions") + except ValueError: + exprs = sh_utility_fat.index.levels[-1] + sh_utility_fat = pd.DataFrame( + sh_utility_fat.values.reshape(-1, len(exprs)), + index=sh_utility_fat.index[:: len(exprs)].droplevel(-1), + columns=exprs, + ) + else: + sh_utility_fat = sh_utility_fat.droplevel(0, axis=1) + sh_utility_fat.add_prefix("SH:") + sh_utility_fat_coef = sh_utility_fat * spec.iloc[:, 0].values.reshape(1, -1) + sh_utility_fat_coef.columns = [ + f"{i} * ({j})" + for i, j in zip(sh_utility_fat_coef.columns, spec.iloc[:, 0].values) + ] + if utilities.shape[0] > trace_rows.shape[0]: + trace_rows_ = np.repeat( + trace_rows, utilities.shape[0] // trace_rows.shape[0] + ) + else: + trace_rows_ = trace_rows + if trace_eval_results is None: + trace_eval_results = pd.concat( + [ + sh_utility_fat, + sh_utility_fat_coef, + utilities.utility[trace_rows_] + .rename("total utility") + .to_frame() + .set_index(sh_utility_fat.index), + ], + axis=1, + ) + try: + trace_eval_results.index = df[trace_rows].index + except ValueError: + pass + chunk.log_df(trace_label, "eval.trace_eval_results", trace_eval_results) + else: + # in test mode, trace from non-sharrow exists + trace_eval_results = pd.concat( + [ + trace_eval_results.reset_index(drop=True), + sh_utility_fat.reset_index(drop=True), + sh_utility_fat_coef.reset_index(drop=True), + utilities.utility[trace_rows_] + .rename("total utility") + .reset_index(drop=True), + ], + axis=1, + ) + trace_eval_results.index = df[trace_rows].index + chunk.log_df(trace_label, "eval.trace_eval_results", trace_eval_results) + + # sh_utility_fat1 = np.dot(sh_utility_fat, spec.values) + # sh_utility_fat2 = sh_flow.dot( + # source=sh_flow.tree.replace_datasets( + # df=df.iloc[trace_rows], + # ), + # coefficients=spec.values.astype(np.float32), + # dtype=np.float32, + # ) + timelogger.mark("sharrow interact trace", True, logger, trace_label) + + if sharrow_enabled == "test": + + try: + if sh_util is not None: + np.testing.assert_allclose( + sh_util.reshape(utilities.values.shape), + utilities.values, + rtol=1e-2, + atol=0, + err_msg="utility not aligned", + verbose=True, + ) + except AssertionError as err: + print(err) + misses = np.where( + ~np.isclose(sh_util, utilities.values, rtol=1e-2, atol=0) + ) + _sh_util_miss1 = sh_util[tuple(m[0] for m in misses)] + _u_miss1 = utilities.values[tuple(m[0] for m in misses)] + diff = _sh_util_miss1 - _u_miss1 + if len(misses[0]) > sh_util.size * 0.01: + print("big problem") + if "nan location mismatch" in str(err): + print("nan location mismatch sh_util") + print(np.where(np.isnan(sh_util))) + print("nan location mismatch legacy util") + print(np.where(np.isnan(utilities.values))) + print("misses =>", misses) + j = 0 + while j < len(misses[0]): + print( + f"miss {j} {tuple(m[j] for m in misses)}:", + sh_util[tuple(m[j] for m in misses)], + "!=", + utilities.values[tuple(m[j] for m in misses)], + ) + j += 1 + if j > 10: + break + + re_trace = misses[0] + retrace_eval_data = {} + retrace_eval_parts = {} + re_trace_df = df.iloc[re_trace] + + for expr, label, coefficient in zip(exprs, labels, spec.iloc[:, 0]): + if expr.startswith("_"): + target = expr[: expr.index("@")] + rhs = expr[expr.index("@") + 1 :] + v = to_series(eval(rhs, globals(), locals_d)) + locals_d[target] = v + if trace_eval_results is not None: + trace_eval_results[expr] = v.iloc[re_trace] + continue + if expr.startswith("@"): + v = to_series(eval(expr[1:], globals(), locals_d)) + else: + v = df.eval(expr) + if check_for_variability and v.std() == 0: + logger.info( + "%s: no variability (%s) in: %s" + % (trace_label, v.iloc[0], expr) + ) + no_variability += 1 + retrace_eval_data[expr] = v.iloc[re_trace] + k = "partial utility (coefficient = %s) for %s" % ( + coefficient, + expr, + ) + retrace_eval_parts[k] = (v.iloc[re_trace] * coefficient).astype( + "float" + ) + retrace_eval_data_ = pd.concat(retrace_eval_data, axis=1) + retrace_eval_parts_ = pd.concat(retrace_eval_parts, axis=1) - if trace_eval_results is not None: - trace_eval_results["total utility"] = utilities.utility[trace_rows] + re_sh_flow_load = sh_flow.load( + dtype=np.float32, + ) + re_sh_flow_load_ = re_sh_flow_load[re_trace] + + look_for_problems_here = np.where( + ~np.isclose( + re_sh_flow_load_[ + :, ~spec.index.get_level_values(0).str.startswith("_") + ], + retrace_eval_data_.values.astype(np.float32), + ) + ) - trace_eval_results = pd.DataFrame.from_dict(trace_eval_results) - trace_eval_results.index = df[trace_rows].index + raise # enter debugger now to see what's up + timelogger.mark("sharrow interact test", True, logger, trace_label) - # add df columns to trace_results - trace_eval_results = pd.concat([df[trace_rows], trace_eval_results], axis=1) - chunk.log_df(trace_label, "eval.trace_eval_results", trace_eval_results) + logger.info(f"utilities.dtypes {trace_label}\n{utilities.dtypes}") + end_time = time.time() - chunk.log_df(trace_label, "v", None) - chunk.log_df(trace_label, "eval.utilities", None) # out of out hands... - chunk.log_df(trace_label, "eval.trace_eval_results", None) + timelogger.summary(logger, "TIMING interact_simulate.eval_utils") + logger.info( + f"interact_simulate.eval_utils runtime: {timedelta(seconds=end_time - start_time)} {trace_label}" + ) return utilities, trace_eval_results @@ -351,66 +695,149 @@ def _interaction_simulate( alt_index_id = estimator.get_alt_id() if estimator else None chooser_index_id = ALT_CHOOSER_ID if log_alt_losers else None - interaction_df = logit.interaction_dataset( - choosers, - alternatives, - sample_size, - alt_index_id=alt_index_id, - chooser_index_id=chooser_index_id, - ) - chunk.log_df(trace_label, "interaction_df", interaction_df) + sharrow_enabled = config.setting("sharrow", False) + interaction_utilities = None - if skims is not None: - simulate.set_skim_wrapper_targets(interaction_df, skims) + if ( + sharrow_enabled + and skims is None + and not have_trace_targets + and sample_size == len(alternatives) + ): + # no need to create the merged interaction dataset + # TODO: can we still do this if skims is not None? - # evaluate expressions from the spec multiply by coefficients and sum - # spec is df with one row per spec expression and one col with utility coefficient - # column names of model_design match spec index values - # utilities has utility value for element in the cross product of choosers and alternatives - # interaction_utilities is a df with one utility column and one row per row in model_design - if have_trace_targets: - trace_rows, trace_ids = tracing.interaction_trace_rows( - interaction_df, choosers, sample_size - ) + # TODO: re-enable tracing for sharrow so have_trace_targets can be True + trace_rows = trace_ids = None - tracing.trace_df( - interaction_df[trace_rows], - tracing.extend_trace_label(trace_label, "interaction_df"), - slicer="NONE", - transpose=False, + interaction_utilities, trace_eval_results = eval_interaction_utilities( + spec, + choosers, + locals_d, + trace_label, + trace_rows, + estimator=estimator, + log_alt_losers=log_alt_losers, + extra_data=alternatives, ) - else: - trace_rows = trace_ids = None - interaction_utilities, trace_eval_results = eval_interaction_utilities( - spec, - interaction_df, - locals_d, - trace_label, - trace_rows, - estimator=estimator, - log_alt_losers=log_alt_losers, - ) - chunk.log_df(trace_label, "interaction_utilities", interaction_utilities) - # print(f"interaction_df {interaction_df.shape}") - # print(f"interaction_utilities {interaction_utilities.shape}") + # set this index here as this is how later code extracts the chosen alt id's + interaction_utilities.index = np.tile(alternatives.index, len(choosers)) - del interaction_df - chunk.log_df(trace_label, "interaction_df", None) + chunk.log_df(trace_label, "interaction_utilities", interaction_utilities) + # mem.trace_memory_info(f"{trace_label}.init interaction_utilities sh", force_garbage_collect=True) + if sharrow_enabled == "test" or True: + interaction_utilities_sh, trace_eval_results_sh = ( + interaction_utilities, + trace_eval_results, + ) + else: + interaction_utilities_sh = trace_eval_results_sh = None - if have_trace_targets: - tracing.trace_interaction_eval_results( - trace_eval_results, - trace_ids, - tracing.extend_trace_label(trace_label, "eval"), + else: + interaction_utilities_sh = trace_eval_results_sh = None + + if ( + not sharrow_enabled + or (sharrow_enabled == "test") + or interaction_utilities is None + ): + + interaction_df = logit.interaction_dataset( + choosers, + alternatives, + sample_size, + alt_index_id=alt_index_id, + chooser_index_id=chooser_index_id, ) + chunk.log_df(trace_label, "interaction_df", interaction_df) + + if skims is not None: + simulate.set_skim_wrapper_targets(interaction_df, skims) + + # evaluate expressions from the spec multiply by coefficients and sum + # spec is df with one row per spec expression and one col with utility coefficient + # column names of model_design match spec index values + # utilities has utility value for element in the cross product of choosers and alternatives + # interaction_utilities is a df with one utility column and one row per row in model_design + if have_trace_targets: + trace_rows, trace_ids = tracing.interaction_trace_rows( + interaction_df, choosers, sample_size + ) - tracing.trace_df( - interaction_utilities[trace_rows], - tracing.extend_trace_label(trace_label, "interaction_utils"), - slicer="NONE", - transpose=False, + tracing.trace_df( + interaction_df[trace_rows], + tracing.extend_trace_label(trace_label, "interaction_df"), + slicer="NONE", + transpose=False, + ) + else: + trace_rows = trace_ids = None + + interaction_utilities, trace_eval_results = eval_interaction_utilities( + spec, + interaction_df, + locals_d, + trace_label, + trace_rows, + estimator=estimator, + log_alt_losers=log_alt_losers, ) + chunk.log_df(trace_label, "interaction_utilities", interaction_utilities) + # mem.trace_memory_info(f"{trace_label}.init interaction_utilities", force_garbage_collect=True) + + # print(f"interaction_df {interaction_df.shape}") + # print(f"interaction_utilities {interaction_utilities.shape}") + + del interaction_df + chunk.log_df(trace_label, "interaction_df", None) + + if have_trace_targets: + tracing.trace_interaction_eval_results( + trace_eval_results, + trace_ids, + tracing.extend_trace_label(trace_label, "eval"), + ) + + tracing.trace_df( + interaction_utilities[trace_rows], + tracing.extend_trace_label(trace_label, "interaction_utils"), + slicer="NONE", + transpose=False, + ) + + if sharrow_enabled == "test": + try: + if interaction_utilities_sh is not None: + np.testing.assert_allclose( + interaction_utilities_sh.values.reshape( + interaction_utilities.values.shape + ), + interaction_utilities.values, + rtol=1e-2, + atol=0, + err_msg="utility not aligned", + verbose=True, + ) + except AssertionError as err: + print(err) + misses = np.where( + ~np.isclose( + interaction_utilities_sh.values, + interaction_utilities.values, + rtol=1e-2, + atol=0, + ) + ) + _sh_util_miss1 = interaction_utilities_sh.values[ + tuple(m[0] for m in misses) + ] + _u_miss1 = interaction_utilities.values[tuple(m[0] for m in misses)] + diff = _sh_util_miss1 - _u_miss1 + if len(misses[0]) > interaction_utilities_sh.values.size * 0.01: + print("big problem") + print(misses) + raise # reshape utilities (one utility column and one row per row in model_design) # to a dataframe with one row per chooser and one column per alternative @@ -569,7 +996,7 @@ def interaction_simulate( result_list.append(choices) - chunk.log_df(trace_label, f"result_list", result_list) + chunk.log_df(trace_label, "result_list", result_list) # FIXME: this will require 2X RAM # if necessary, could append to hdf5 store on disk: diff --git a/activitysim/core/logit.py b/activitysim/core/logit.py index 784e073f86..ff3128450d 100644 --- a/activitysim/core/logit.py +++ b/activitysim/core/logit.py @@ -7,6 +7,7 @@ import pandas as pd from . import config, pipeline, tracing +from .choosing import choice_maker logger = logging.getLogger(__name__) @@ -156,26 +157,35 @@ def utils_to_probs( # fixme - conversion to float not needed in either case? # utils_arr = utils.values.astype('float') utils_arr = utils.values + + if utils_arr.dtype == np.float32 and utils_arr.max() > 85: + # exponentiated utils will overflow, downshift them + utils_arr -= utils_arr.max(1, keepdims=True) + if not exponentiated: + # TODO: reduce memory usage by exponentiating in-place. + # but first we need to make sure the raw utilities + # are not needed elsewhere and overwriting won't hurt. + # try: + # np.exp(utils_arr, out=utils_arr) + # except TypeError: + # utils_arr = np.exp(utils_arr) utils_arr = np.exp(utils_arr) - np.clip(utils_arr, EXP_UTIL_MIN, EXP_UTIL_MAX, out=utils_arr) - - # FIXME - utils_arr = np.where(utils_arr == EXP_UTIL_MIN, 0.0, utils_arr) + np.putmask(utils_arr, utils_arr <= EXP_UTIL_MIN, 0) arr_sum = utils_arr.sum(axis=1) - zero_probs = arr_sum == 0.0 - if zero_probs.any() and not allow_zero_probs: - - report_bad_choices( - zero_probs, - utils, - trace_label=tracing.extend_trace_label(trace_label, "zero_prob_utils"), - msg="all probabilities are zero", - trace_choosers=trace_choosers, - ) + if not allow_zero_probs: + zero_probs = arr_sum == 0.0 + if zero_probs.any(): + report_bad_choices( + zero_probs, + utils, + trace_label=tracing.extend_trace_label(trace_label, "zero_prob_utils"), + msg="all probabilities are zero", + trace_choosers=trace_choosers, + ) inf_utils = np.isinf(arr_sum) if inf_utils.any(): @@ -195,7 +205,7 @@ def utils_to_probs( np.divide(utils_arr, arr_sum.reshape(len(utils_arr), 1), out=utils_arr) # if allow_zero_probs, this will cause EXP_UTIL_MIN util rows to have all zero probabilities - utils_arr[np.isnan(utils_arr)] = PROB_MIN + np.putmask(utils_arr, np.isnan(utils_arr), PROB_MIN) np.clip(utils_arr, PROB_MIN, PROB_MAX, out=utils_arr) @@ -251,13 +261,7 @@ def make_choices(probs, trace_label=None, trace_choosers=None, allow_bad_probs=F rands = pipeline.get_rn_generator().random_for_df(probs) - probs_arr = probs.values.cumsum(axis=1) - rands - - # rows, cols = np.where(probs_arr > 0) - # choices = [s.iat[0] for _, s in pd.Series(cols).groupby(rows)] - choices = np.argmax(probs_arr > 0.0, axis=1) - - choices = pd.Series(choices, index=probs.index) + choices = pd.Series(choice_maker(probs.values, rands), index=probs.index) rands = pd.Series(np.asanyarray(rands).flatten(), index=probs.index) diff --git a/activitysim/core/los.py b/activitysim/core/los.py index b274941f1d..7304d5b0a8 100644 --- a/activitysim/core/los.py +++ b/activitysim/core/los.py @@ -2,21 +2,14 @@ # See full license in LICENSE.txt. import logging -import os import warnings import numpy as np import pandas as pd -from activitysim.core import ( - config, - inject, - mem, - pathbuilder, - skim_dictionary, - tracing, - util, -) +from activitysim.core import skim_dataset # noqa: F401 +from activitysim.core import config, inject, pathbuilder, skim_dictionary, tracing, util +from activitysim.core.cleaning import recode_based_on_table from activitysim.core.skim_dict_factory import MemMapSkimFactory, NumpyArraySkimFactory from activitysim.core.skim_dictionary import NOT_IN_SKIM_ZONE_ID @@ -137,15 +130,19 @@ def setting(self, keys, default=""): s = self.los_settings for key in key_list[:-1]: s = s.get(key) - assert isinstance( - s, dict - ), f"expected key '{key}' not found in '{keys}' in {self.los_settings_file_name}" + if default == "": + assert isinstance( + s, dict + ), f"expected key '{key}' not found in '{keys}' in {self.los_settings_file_name}" key = key_list[-1] # last key if default == "": assert ( key in s ), f"Expected setting {keys} not found in in {LOS_SETTINGS_FILE_NAME}" - return s.get(key, default) + if isinstance(s, dict): + return s.get(key, default) + else: + return default def load_settings(self): """ @@ -276,6 +273,14 @@ def load_data(self): by="MAZ" ) # only fields we need + # recode MAZs if needed + self.maz_taz_df["MAZ"] = recode_based_on_table( + self.maz_taz_df["MAZ"], "land_use" + ) + self.maz_taz_df["TAZ"] = recode_based_on_table( + self.maz_taz_df["TAZ"], "land_use_taz" + ) + self.maz_ceiling = self.maz_taz_df.MAZ.max() + 1 # maz_to_maz_df @@ -325,6 +330,9 @@ def load_data(self): file_name = maz_to_tap_settings["table"] df = pd.read_csv(config.data_file_path(file_name, mandatory=True)) + # recode MAZs if needed + df["MAZ"] = recode_based_on_table(df["MAZ"], "land_use") + # trim tap set # if provided, use tap_line_distance_col together with tap_lines table to trim the near tap set # to only include the nearest tap to origin when more than one tap serves the same line @@ -414,38 +422,57 @@ def load_data(self): self.maz_to_tap_dfs[mode] = df # create taz skim dict - assert "taz" not in self.skim_dicts - self.skim_dicts["taz"] = self.create_skim_dict("taz") - - # make sure skim has all taz_ids - # FIXME - weird that there is no list of tazs? + if not config.setting("sharrow", False): + assert "taz" not in self.skim_dicts + # If offset_preprocessing was completed, then TAZ values + # will be pre-offset and there's no need to re-offset them. + if config.setting("offset_preprocessing", False): + _override_offset_int = 0 + else: + _override_offset_int = None + self.skim_dicts["taz"] = self.create_skim_dict( + "taz", _override_offset_int=_override_offset_int + ) + # make sure skim has all taz_ids + # FIXME - weird that there is no list of tazs? + else: + self.skim_dicts["taz"] = self.get_skim_dict("taz") # create MazSkimDict facade if self.zone_system in [TWO_ZONE, THREE_ZONE]: - # create MazSkimDict facade skim_dict - # (must have already loaded dependencies: taz skim_dict, maz_to_maz_df, and maz_taz_df) - assert "maz" not in self.skim_dicts - maz_skim_dict = self.create_skim_dict("maz") - self.skim_dicts["maz"] = maz_skim_dict - - # make sure skim has all maz_ids - assert not ( - maz_skim_dict.offset_mapper.map(self.maz_taz_df["MAZ"].values) - == NOT_IN_SKIM_ZONE_ID - ).any() + if not config.setting("sharrow", False): + # create MazSkimDict facade skim_dict + # (must have already loaded dependencies: taz skim_dict, maz_to_maz_df, and maz_taz_df) + assert "maz" not in self.skim_dicts + maz_skim_dict = self.create_skim_dict("maz") + self.skim_dicts["maz"] = maz_skim_dict + + # make sure skim has all maz_ids + assert not ( + maz_skim_dict.offset_mapper.map(self.maz_taz_df["MAZ"].values) + == NOT_IN_SKIM_ZONE_ID + ).any(), ( + "every MAZ in the MAZ-to-TAZ mapping must map to a TAZ that exists" + ) + else: + self.skim_dicts["maz"] = self.get_skim_dict("maz") + # TODO:SHARROW: make sure skim has all maz_ids # create tap skim dict if self.zone_system == THREE_ZONE: - assert "tap" not in self.skim_dicts - tap_skim_dict = self.create_skim_dict("tap") - self.skim_dicts["tap"] = tap_skim_dict - # make sure skim has all tap_ids - assert not ( - tap_skim_dict.offset_mapper.map(self.tap_df["TAP"].values) - == NOT_IN_SKIM_ZONE_ID - ).any() + if not config.setting("sharrow", False): + assert "tap" not in self.skim_dicts + tap_skim_dict = self.create_skim_dict("tap") + self.skim_dicts["tap"] = tap_skim_dict + # make sure skim has all tap_ids + assert not ( + tap_skim_dict.offset_mapper.map(self.tap_df["TAP"].values) + == NOT_IN_SKIM_ZONE_ID + ).any() + else: + self.skim_dicts["tap"] = self.get_skim_dict("tap") - def create_skim_dict(self, skim_tag): + def create_skim_dict(self, skim_tag, _override_offset_int=None): """ Create a new SkimDict of type specified by skim_tag (e.g. 'taz', 'maz' or 'tap') @@ -477,6 +504,11 @@ def create_skim_dict(self, skim_tag): logger.debug(f"create_skim_dict {skim_tag} omx_shape {skim_dict.omx_shape}") + if _override_offset_int is not None: + skim_dict.offset_mapper.set_offset_int( + _override_offset_int + ) # default is -1 + return skim_dict def omx_file_names(self, skim_tag): @@ -492,9 +524,66 @@ def omx_file_names(self, skim_tag): list of str """ file_names = self.setting(f"{skim_tag}_skims") + if isinstance(file_names, dict): + for i in ("file", "files", "omx"): + if i in file_names: + file_names = file_names[i] + break + if isinstance(file_names, dict): + raise ValueError( + f"must specify `{skim_tag}_skims.file` in network_los settings file" + ) file_names = [file_names] if isinstance(file_names, str) else file_names return file_names + def zarr_file_name(self, skim_tag): + """ + Return zarr directory name from network_los settings file for the specified skim_tag (e.g. 'taz') + + Parameters + ---------- + skim_tag: str (e.g. 'taz') + + Returns + ------- + list of str + """ + skim_setting = self.setting(f"{skim_tag}_skims") + if isinstance(skim_setting, dict): + return skim_setting.get("zarr", None) + else: + return None + + def zarr_pre_encoding(self, skim_tag): + """ + Return digital encoding pre-processing before writing to zarr for the specified skim_tag (e.g. 'taz') + + Parameters + ---------- + skim_tag: str (e.g. 'taz') + + Returns + ------- + list or None + """ + skim_setting = self.setting(f"{skim_tag}_skims") + if isinstance(skim_setting, dict): + return skim_setting.get("zarr-digital-encoding", None) + else: + return None + + def skim_backing_store(self, skim_tag): + name = self.setting("name", "unnamed") + return self.setting( + f"{skim_tag}_skims.backend", f"shared_memory_{skim_tag}_{name}" + ) + + def skim_max_float_precision(self, skim_tag): + return self.setting(f"{skim_tag}_skims.max_float_precision", 32) + + def skim_digital_encoding(self, skim_tag): + return self.setting(f"{skim_tag}_skims.digital_encoding", []) + def multiprocess(self): """ return True if this is a multiprocessing run (even if it is a main or single-process subprocess) @@ -584,11 +673,31 @@ def get_skim_dict(self, skim_tag): ------- SkimDict or subclass (e.g. MazSkimDict) """ - - assert ( - skim_tag in self.skim_dicts - ), f"network_los.get_skim_dict: skim tag '{skim_tag}' not in skim_dicts" - return self.skim_dicts[skim_tag] + # TODO:SHARROW: TAZ and MAZ are the same + sharrow_enabled = config.setting("sharrow", False) + if sharrow_enabled and skim_tag in ("taz", "maz"): + skim_dataset = inject.get_injectable("skim_dataset") + from .skim_dataset import SkimDataset + + if skim_tag == "maz": + return SkimDataset(skim_dataset) + else: + dropdims = ["omaz", "dmaz"] + skim_dataset = skim_dataset.drop_dims(dropdims, errors="ignore") + for dd in dropdims: + if f"dim_redirection_{dd}" in skim_dataset.attrs: + del skim_dataset.attrs[f"dim_redirection_{dd}"] + return SkimDataset(skim_dataset) + elif sharrow_enabled and skim_tag in ("tap"): + tap_dataset = inject.get_injectable("tap_dataset") + from .skim_dataset import SkimDataset + + return SkimDataset(tap_dataset) + else: + assert ( + skim_tag in self.skim_dicts + ), f"network_los.get_skim_dict: skim tag '{skim_tag}' not in skim_dicts" + return self.skim_dicts[skim_tag] def get_default_skim_dict(self): """ @@ -601,6 +710,7 @@ def get_default_skim_dict(self): if self.zone_system == ONE_ZONE: return self.get_skim_dict("taz") else: + # TODO:SHARROW: taz and maz are the same return self.get_skim_dict("maz") def get_mazpairs(self, omaz, dmaz, attribute): @@ -651,9 +761,36 @@ def get_tappairs3d(self, otap, dtap, dim3, key): ------- Numpy.ndarray: list of tap skim values for odt tuples """ + tap_skim = self.get_skim_dict("tap") + + if isinstance(tap_skim, skim_dictionary.SkimDict): + return tap_skim.lookup_3d(otap, dtap, dim3, key) + elif isinstance(dim3, str): + s = ( + tap_skim.dataset[[key]] + .sel(time_period=dim3) + .at( + otap=otap.values, + dtap=dtap.values, + _name=key, + ) + ) + elif dim3.dtype.kind == "i": + s = tap_skim.dataset.at( + otap=otap.values, + dtap=dtap.values, + time_period=tap_skim.dataset.time_period.values[dim3], + _name=key, + ) + else: + s = tap_skim.dataset.at( + otap=otap.values, + dtap=dtap.values, + time_period=dim3, + _name=key, + ) - s = self.get_skim_dict("tap").lookup_3d(otap, dtap, dim3, key) - return s + return s.values def skim_time_period_label(self, time_period): """ @@ -665,7 +802,7 @@ def skim_time_period_label(self, time_period): Returns ------- - numpy.array + pandas Series string time period labels """ @@ -683,22 +820,42 @@ def skim_time_period_label(self, time_period): assert 0 == model_time_window_min % period_minutes total_periods = model_time_window_min / period_minutes - bins = ( - np.digitize( - [np.array(time_period) % total_periods], + # FIXME - eventually test and use np version always? + if np.isscalar(time_period): + bin = ( + np.digitize( + [time_period % total_periods], + self.skim_time_periods["periods"], + right=True, + )[0] + - 1 + ) + result = self.skim_time_periods["labels"][bin] + else: + result = pd.cut( + time_period, self.skim_time_periods["periods"], - right=True, - )[0] - - 1 - ) - return np.array(self.skim_time_periods["labels"])[bins] + labels=self.skim_time_periods["labels"], + ordered=False, + ).astype(str) + + return result def get_tazs(self): # FIXME - should compute on init? if self.zone_system == ONE_ZONE: tazs = inject.get_table("land_use").index.values else: - tazs = self.maz_taz_df.TAZ.unique() + try: + land_use_taz = inject.get_table("land_use_taz").to_frame() + except (RuntimeError, KeyError): + # land_use_taz is missing, use fallback + tazs = self.maz_taz_df.TAZ.unique() + else: + if "_original_TAZ" in land_use_taz: + tazs = land_use_taz["_original_TAZ"].values + else: + tazs = self.maz_taz_df.TAZ.unique() assert isinstance(tazs, np.ndarray) return tazs @@ -715,3 +872,49 @@ def get_taps(self): taps = self.tap_df.TAP.values assert isinstance(taps, np.ndarray) return taps + + @property + def get_maz_to_taz_series(self): + """ + pd.Series: Index is the MAZ, value is the corresponding TAZ + """ + sharrow_enabled = config.setting("sharrow", False) + if sharrow_enabled: + # FIXME:SHARROW - this assumes that both MAZ and TAZ have been recoded to + # zero-based indexes, but what if that was not done? + # Should we check it and error out here or bravely march forward? + skim_dataset = inject.get_injectable("skim_dataset") + maz_to_taz = skim_dataset["_digitized_otaz_of_omaz"].to_series() + else: + maz_to_taz = self.maz_taz_df[["MAZ", "TAZ"]].set_index("MAZ").TAZ + return maz_to_taz + + def map_maz_to_taz(self, s): + """ + Convert MAZ's to TAZ's + + Parameters + ---------- + s : Array-like + Integer MAZ values + + Returns + ------- + pd.Series + Integer TAZ values + """ + if not isinstance(s, (pd.Series, pd.Index)): + s = pd.Series(s) + input_was_series = False + else: + input_was_series = True + out = s.map(self.get_maz_to_taz_series) + if np.issubdtype(out, np.floating): + if out.isna().any(): + raise KeyError("failed in mapping MAZ to TAZ") + else: + out = out.astype(np.int32) + if input_was_series: + return out + else: + return out.to_numpy() diff --git a/activitysim/core/mem.py b/activitysim/core/mem.py index 845ea35542..013b7627a9 100644 --- a/activitysim/core/mem.py +++ b/activitysim/core/mem.py @@ -174,7 +174,7 @@ def log_global_hwm(): ) -def trace_memory_info(event, trace_ticks=0): +def trace_memory_info(event, trace_ticks=0, force_garbage_collect=False): global MEM_TICK @@ -183,6 +183,14 @@ def trace_memory_info(event, trace_ticks=0): return MEM_TICK = tick + if force_garbage_collect: + was_disabled = not gc.isenabled() + if was_disabled: + gc.enable() + gc.collect() + if was_disabled: + gc.disable() + process_name = multiprocessing.current_process().name pid = os.getpid() @@ -276,6 +284,11 @@ def shared_memory_size(data_buffers=None): data_buffers = inject.get_injectable("data_buffers", {}) for k, data_buffer in data_buffers.items(): + if isinstance(data_buffer, str) and data_buffer.startswith("sh.Dataset:"): + from sharrow import Dataset + + shared_size += Dataset.shm.preload_shared_memory_size(data_buffer[11:]) + continue try: obj = data_buffer.get_obj() except Exception: diff --git a/activitysim/core/memory_sidecar.py b/activitysim/core/memory_sidecar.py new file mode 100644 index 0000000000..4e314efe9f --- /dev/null +++ b/activitysim/core/memory_sidecar.py @@ -0,0 +1,130 @@ +import datetime +import os +import time +from multiprocessing import Pipe, Process, Queue + +import psutil + + +def record_memory_usage( + logstream, event="", event_idx=-1, measure_uss=False, measure_cpu=False, pid=None +): + + if pid is None: + pid = os.getpid() + current_process = psutil.Process(pid) + with current_process.oneshot(): + process_name = current_process.name() + + if measure_uss: + try: + info = current_process.memory_full_info() + uss = info.uss + except (PermissionError, psutil.AccessDenied, RuntimeError): + info = current_process.memory_info() + uss = 0 + else: + info = current_process.memory_info() + uss = 0 + + if measure_cpu: + cpu_pct = current_process.cpu_percent() + else: + cpu_pct = -1 + + full_rss = rss = info.rss + + num_children = 0 + for child in current_process.children(recursive=True): + try: + child_info = child.memory_info() + full_rss += child_info.rss + num_children += 1 + except (psutil.NoSuchProcess, psutil.AccessDenied) as e: + pass + + timestamp = datetime.datetime.now().strftime("%Y/%m/%d %H:%M:%S.%f") # sortable + + print( + f"{process_name}," + f"{pid}," + f"{int(rss)}," + f"{int(full_rss)}," + f"{int(uss)}," + f"{cpu_pct}," + f"{event_idx}," + f"{event}," + f"{num_children}," + f"{timestamp}", + file=logstream, + ) + + +def monitor_memory_usage( + pid, + conn, + interval=0.5, + flush_interval=5, + filename="/tmp/sidecar.csv", + measure_uss=True, + measure_cpu=True, +): + event = "" + event_idx = 0 + last_flush = time.time() + if measure_cpu: + psutil.cpu_percent() + with open(filename, "w") as stream: + MEM_LOG_HEADER = ( + "process,pid,rss,full_rss,uss,cpu,event_idx,event,children,time" + ) + print(MEM_LOG_HEADER, file=stream) + while True: + # timestamp = datetime.datetime.now().strftime("%Y/%m/%d %H:%M:%S.%f") # sortable + # print(f"{timestamp} [{event}]", file=stream) + record_memory_usage( + stream, + event=event, + event_idx=event_idx, + measure_uss=measure_uss, + measure_cpu=measure_cpu, + pid=pid, + ) + if conn.poll(interval): + event_ = conn.recv() + if event_ != event: + event_idx += 1 + event = event_ + else: + pass + now = time.time() + if now > last_flush + flush_interval: + stream.flush() + last_flush = now + if event == "STOP": + stream.flush() + break + + +class MemorySidecar: + def __init__(self, filename="/tmp/sidecar.csv"): + self.local_conn, child_conn = Pipe() + self.sidecar_process = Process( + target=monitor_memory_usage, + args=(os.getpid(), child_conn), + kwargs=dict(filename=filename), + ) + self.sidecar_process.start() + + def stop(self): + self.set_event("STOP") + self.sidecar_process.join(timeout=5) + if self.sidecar_process.exitcode is None: + self.sidecar_process.kill() + print("memory sidecar stopped") + + def set_event(self, event): + try: + self.local_conn.send(str(event)) + except BrokenPipeError: + pass diff --git a/activitysim/core/mp_tasks.py b/activitysim/core/mp_tasks.py index 1d41c7c96f..c0aab474cd 100644 --- a/activitysim/core/mp_tasks.py +++ b/activitysim/core/mp_tasks.py @@ -1,5 +1,6 @@ # ActivitySim # See full license in LICENSE.txt. +import importlib import logging import multiprocessing import os @@ -381,7 +382,7 @@ def build_slice_rules(slice_info, pipeline_tables): """ slicer_table_names = slice_info["tables"] - slicer_table_exceptions = slice_info.get("except", []) + slicer_table_exceptions = slice_info.get("exclude", slice_info.get("except", [])) primary_slicer = slicer_table_names[0] # - ensure that tables listed in slice_info appear in correct order and before any others @@ -733,15 +734,29 @@ def setup_injectables_and_logging(injectables, locutor=True): # other callers (e.g. piopulationsim) will have to arrange to register their own steps and injectables # (presumably) in a custom run_simulation.py instead of using the 'activitysim run' command if not inject.is_injectable("preload_injectables"): - from activitysim import ( # register abm steps and other abm-specific injectables - abm, - ) + # register abm steps and other abm-specific injectables + from activitysim import abm # noqa: F401 try: for k, v in injectables.items(): inject.add_injectable(k, v) + # re-import extension modules to register injectables + ext = inject.get_injectable("imported_extensions", default=()) + for e in ext: + basepath, extpath = os.path.split(e) + if not basepath: + basepath = "." + sys.path.insert(0, basepath) + try: + importlib.import_module(e) + except ImportError as err: + logger.exception("ImportError") + raise + finally: + del sys.path[0] + inject.add_injectable("is_sub_task", True) inject.add_injectable("locutor", locutor) @@ -1385,15 +1400,18 @@ def skip_phase(phase): def find_breadcrumb(crumb, default=None): return old_breadcrumbs.get(step_name, {}).get(crumb, default) + sharrow_enabled = config.setting("sharrow", False) + # - allocate shared data shared_data_buffers = {} mem.trace_memory_info("allocate_shared_skim_buffer.before") t0 = tracing.print_elapsed_time() - shared_data_buffers.update(allocate_shared_skim_buffers()) - t0 = tracing.print_elapsed_time("allocate shared skim buffer", t0) - mem.trace_memory_info("allocate_shared_skim_buffer.completed") + if not sharrow_enabled: + shared_data_buffers.update(allocate_shared_skim_buffers()) + t0 = tracing.print_elapsed_time("allocate shared skim buffer", t0) + mem.trace_memory_info("allocate_shared_skim_buffer.completed") # combine shared_skim_buffer and shared_shadow_pricing_buffer in shared_data_buffer t0 = tracing.print_elapsed_time() @@ -1401,18 +1419,37 @@ def find_breadcrumb(crumb, default=None): t0 = tracing.print_elapsed_time("allocate shared shadow_pricing buffer", t0) mem.trace_memory_info("allocate_shared_shadow_pricing_buffers.completed") + if sharrow_enabled: + start_time = time.time() + shared_data_buffers["skim_dataset"] = "sh.Dataset:skim_dataset" + + # Loading skim_dataset must be done in the main process, not a subprocess, + # so that this min process can hold on to the shared memory and then cleanly + # release it on exit. + from . import flow # make injectable known # noqa: F401 + + inject.get_injectable("skim_dataset") + + t0 = tracing.print_elapsed_time("setup skim_dataset", t0) + mem.trace_memory_info("skim_dataset.completed") + tracing.log_runtime("mp_setup_skims", start_time=start_time, force=True) + # - mp_setup_skims - if len(shared_data_buffers) > 0: - run_sub_task( - multiprocessing.Process( - target=mp_setup_skims, - name="mp_setup_skims", - args=(injectables,), - kwargs=shared_data_buffers, + if not sharrow_enabled: + if len(shared_data_buffers) > 0: + start_time = time.time() + run_sub_task( + multiprocessing.Process( + target=mp_setup_skims, + name="mp_setup_skims", + args=(injectables,), + kwargs=shared_data_buffers, + ) ) - ) - t0 = tracing.print_elapsed_time("setup shared_data_buffers", t0) - mem.trace_memory_info("mp_setup_skims.completed") + + t0 = tracing.print_elapsed_time("setup shared_data_buffers", t0) + mem.trace_memory_info("mp_setup_skims.completed") + tracing.log_runtime("mp_setup_skims", start_time=start_time, force=True) # - for each step in run list for step_info in run_list["multiprocess_steps"]: @@ -1429,6 +1466,7 @@ def find_breadcrumb(crumb, default=None): # - mp_apportion_pipeline if not skip_phase("apportion") and num_processes > 1: + start_time = time.time() run_sub_task( multiprocessing.Process( target=mp_apportion_pipeline, @@ -1436,6 +1474,9 @@ def find_breadcrumb(crumb, default=None): args=(injectables, sub_proc_names, step_info), ) ) + tracing.log_runtime( + "%s_apportion" % step_name, start_time=start_time, force=True + ) drop_breadcrumb(step_name, "apportion") # - run_sub_simulations @@ -1463,6 +1504,7 @@ def find_breadcrumb(crumb, default=None): # - mp_coalesce_pipelines if not skip_phase("coalesce") and num_processes > 1: + start_time = time.time() run_sub_task( multiprocessing.Process( target=mp_coalesce_pipelines, @@ -1470,6 +1512,9 @@ def find_breadcrumb(crumb, default=None): args=(injectables, sub_proc_names, slice_info), ) ) + tracing.log_runtime( + "%s_coalesce" % step_name, start_time=start_time, force=True + ) drop_breadcrumb(step_name, "coalesce") # add checkpoint with final tables even if not intermediate checkpointing @@ -1786,7 +1831,11 @@ def get_run_list(): # - add resume breadcrumbs if resume_after: - breadcrumbs = get_breadcrumbs(run_list) + try: + breadcrumbs = get_breadcrumbs(run_list) + except IOError: # file does not exist, no resume_after is possible + breadcrumbs = None + resume_after = None if breadcrumbs: run_list["breadcrumbs"] = breadcrumbs diff --git a/activitysim/core/pathbuilder.py b/activitysim/core/pathbuilder.py index 91107e08e7..1199f19c57 100644 --- a/activitysim/core/pathbuilder.py +++ b/activitysim/core/pathbuilder.py @@ -55,7 +55,7 @@ def compute_utilities( f"{trace_label} Running compute_utilities with {choosers.shape[0]} choosers" ) - locals_dict = {"np": np, "los": network_los} + locals_dict = {"np": np, "los": network_los, "disable_sharrow": True} locals_dict.update(model_constants) # we don't grok coefficients, but allow them to use constants in spec alt columns @@ -1267,6 +1267,16 @@ def __getitem__(self, path_type): skim: Skim The skim object """ + if self.cache_choices and path_type in self.cache: + # restore out of cache if all logsums are available in cache + # this can happen if the tvpb is called twice for the same thing in a spec + # do we want to allow this? alternatively the onus can be on the + # spec writer not to use them twice + cached = self.cache.get(path_type) + if "logsum" in cached and self.df.index[0] in cached.index: + recalled_logsums = cached.reindex(self.df.index).logsum + if not recalled_logsums.isna().any(): + return recalled_logsums assert self.df is not None, "Call set_df first" assert ( @@ -1308,11 +1318,21 @@ def __getitem__(self, path_type): choices_df = logsum_df[["atap", "btap", "path_set"]] if path_type in self.cache: - assert ( + if ( len(self.cache.get(path_type).index.intersection(logsum_df.index)) == 0 - ) - choices_df = pd.concat([self.cache.get(path_type), choices_df]) + ): + choices_df = pd.concat([self.cache.get(path_type), choices_df]) + else: + intersect = self.cache.get(path_type).index.intersection( + logsum_df.index + ) + choices_df = pd.concat( + [ + self.cache.get(path_type), + choices_df.loc[~choices_df.index.isin(intersect)], + ] + ) self.cache[path_type] = choices_df diff --git a/activitysim/core/pathbuilder_cache.py b/activitysim/core/pathbuilder_cache.py index b5f1089138..1bd2196934 100644 --- a/activitysim/core/pathbuilder_cache.py +++ b/activitysim/core/pathbuilder_cache.py @@ -1,19 +1,16 @@ # ActivitySim # See full license in LICENSE.txt. -import gc as _gc import itertools import logging import multiprocessing import os -import time from builtins import range from contextlib import contextmanager import numpy as np import pandas as pd -import psutil -from activitysim.core import config, inject, simulate, tracing, util +from activitysim.core import config, inject, simulate, util logger = logging.getLogger(__name__) @@ -30,37 +27,38 @@ @contextmanager def memo(tag, console=False, disable_gc=True): - t0 = time.time() - - MEMO_STACK.append(tag) - - gc_was_enabled = _gc.isenabled() - if gc_was_enabled: - _gc.collect() - if disable_gc: - _gc.disable() - - previous_mem = psutil.Process().memory_info().rss - try: - yield - finally: - elapsed_time = time.time() - t0 - - current_mem = psutil.Process().memory_info().rss - marginal_mem = current_mem - previous_mem - mem_str = f"net {util.GB(marginal_mem)} ({util.INT(marginal_mem)}) total {util.GB(current_mem)}" - - if gc_was_enabled and disable_gc: - _gc.enable() - if _gc.isenabled(): - _gc.collect() - - if console: - print(f"MEMO {tag} Time: {util.SEC(elapsed_time)} Memory: {mem_str} ") - else: - logger.debug(f"MEM {tag} {mem_str} in {util.SEC(elapsed_time)}") - - MEMO_STACK.pop() + yield # make this a noop for performance + # t0 = time.time() + # + # MEMO_STACK.append(tag) + # + # gc_was_enabled = _gc.isenabled() + # if gc_was_enabled: + # _gc.collect() + # if disable_gc: + # _gc.disable() + # + # previous_mem = psutil.Process().memory_info().rss + # try: + # yield + # finally: + # elapsed_time = time.time() - t0 + # + # current_mem = psutil.Process().memory_info().rss + # marginal_mem = current_mem - previous_mem + # mem_str = f"net {util.GB(marginal_mem)} ({util.INT(marginal_mem)}) total {util.GB(current_mem)}" + # + # if gc_was_enabled and disable_gc: + # _gc.enable() + # if _gc.isenabled(): + # _gc.collect() + # + # if console: + # print(f"MEMO {tag} Time: {util.SEC(elapsed_time)} Memory: {mem_str} ") + # else: + # logger.debug(f"MEM {tag} {mem_str} in {util.SEC(elapsed_time)}") + # + # MEMO_STACK.pop() class TVPBCache(object): @@ -97,7 +95,20 @@ def cleanup(self): """ if os.path.isfile(self.cache_path): logger.debug(f"deleting cache {self.cache_path}") - os.unlink(self.cache_path) + try: + os.unlink(self.cache_path) + except PermissionError: + # windows may complain if the cache was not completely closed + # in an earlier run, so let's just cache in a new file + n = 0 + while True: + n += 1 + candidate = os.path.join( + config.get_cache_dir(), f"{self.cache_tag}.{n}.mmap" + ) + if not os.path.isfile(candidate): + self.cache_tag = f"{self.cache_tag}.{n}" + break def write_static_cache(self, data): @@ -400,13 +411,19 @@ def get_unique_ids(self, df, scalar_attributes): if name in df: # if there is a column, use it - uid = uid * cardinality + np.asanyarray(df[name].map(ordinalizer)) + if name == "tod" and df[name].dtype.kind == "i": + # when time of day is an integer, assume it is already ordinalized + ticker = np.asanyarray(df[name]) + else: + ticker = np.asanyarray(df[name].map(ordinalizer)) + uid = uid * cardinality + ticker else: # otherwise it should be in scalar_attributes assert ( name in scalar_attributes ), f"attribute '{name}' not found in df.columns or scalar_attributes." - uid = uid * cardinality + ordinalizer.at[scalar_attributes[name]] + ticker = ordinalizer.at[scalar_attributes[name]] + uid = uid * cardinality + ticker return uid diff --git a/activitysim/core/pipeline.py b/activitysim/core/pipeline.py index ad39cdcfed..fe3ceca456 100644 --- a/activitysim/core/pipeline.py +++ b/activitysim/core/pipeline.py @@ -413,6 +413,14 @@ def load_checkpoint(checkpoint_name): # register it as an orca table rewrap(table_name, df) loaded_tables[table_name] = df + if table_name == "land_use" and "_original_zone_id" in df.columns: + # The presence of _original_zone_id indicates this table index was + # decoded to zero-based, so we need to disable offset + # processing for legacy skim access. + # TODO: this "magic" column name should be replaced with a mechanism + # to write and recover particular settings from the pipeline + # store, but we don't have that mechanism yet + config.override_setting("offset_preprocessing", True) # register for tracing in order that tracing.register_traceable_table wants us to register them traceable_tables = inject.get_injectable("traceable_tables", []) @@ -499,7 +507,26 @@ def run_model(model_name): t0 = print_elapsed_time() logger.info(f"#run_model running step {step_name}") - orca.run([step_name]) + instrument = config.setting("instrument", None) + if instrument is not None: + try: + from pyinstrument import Profiler + except ImportError: + instrument = False + if isinstance(instrument, (list, set, tuple)): + if step_name not in instrument: + instrument = False + else: + instrument = True + + if instrument: + with Profiler() as profiler: + orca.run([step_name]) + out_file = config.profiling_file_path(f"{step_name}.html") + with open(out_file, "wt") as f: + f.write(profiler.output_html()) + else: + orca.run([step_name]) t0 = print_elapsed_time( "#run_model completed step '%s'" % model_name, t0, debug=True @@ -543,7 +570,15 @@ def open_pipeline(resume_after=None, mode="a"): # open existing pipeline logger.debug("open_pipeline - open existing pipeline") open_pipeline_store(overwrite=False, mode=mode) - load_checkpoint(resume_after) + try: + load_checkpoint(resume_after) + except KeyError as err: + if "checkpoints" in err.args[0]: + # no checkpoints initialized, fall back to restart + _PIPELINE.last_checkpoint[CHECKPOINT_NAME] = INITIAL_CHECKPOINT_NAME + add_checkpoint(INITIAL_CHECKPOINT_NAME) + else: + raise else: # open new, empty pipeline logger.debug("open_pipeline - new, empty pipeline") @@ -601,7 +636,7 @@ def intermediate_checkpoint(checkpoint_name=None): return checkpoint_name in checkpoints -def run(models, resume_after=None): +def run(models, resume_after=None, memory_sidecar_process=None): """ run the specified list of models, optionally loading checkpoint and resuming after specified checkpoint. @@ -618,6 +653,8 @@ def run(models, resume_after=None): list of model_names resume_after : str or None model_name of checkpoint to load checkpoint and AFTER WHICH to resume model run + memory_sidecar_process : MemorySidecar, optional + Subprocess that monitors memory usage returns: nothing, but with pipeline open @@ -640,18 +677,25 @@ def run(models, resume_after=None): # preload any bulky injectables (e.g. skims) not in pipeline if inject.get_injectable("preload_injectables", None): + if memory_sidecar_process: + memory_sidecar_process.set_event("preload_injectables") t0 = print_elapsed_time("preload_injectables", t0) mem.trace_memory_info("pipeline.run after preload_injectables") t0 = print_elapsed_time() for model in models: + if memory_sidecar_process: + memory_sidecar_process.set_event(model) t1 = print_elapsed_time() run_model(model) mem.trace_memory_info(f"pipeline.run after {model}") tracing.log_runtime(model_name=model, start_time=t1) + if memory_sidecar_process: + memory_sidecar_process.set_event("finalizing") + # add checkpoint with final tables even if not intermediate checkpointing if not intermediate_checkpoint(): add_checkpoint(FINAL_CHECKPOINT_NAME) diff --git a/activitysim/core/random.py b/activitysim/core/random.py index 12527840f4..94491d1c75 100644 --- a/activitysim/core/random.py +++ b/activitysim/core/random.py @@ -247,7 +247,7 @@ def random_for_df(self, df, step_name, n=1): self.row_states.loc[df.index, "offset"] += n return rands - def normal_for_df(self, df, step_name, mu, sigma, lognormal=False): + def normal_for_df(self, df, step_name, mu, sigma, lognormal=False, size=None): """ Return a floating point random number in normal (or lognormal) distribution for each row in df using the appropriate random channel for each row. @@ -296,20 +296,24 @@ def to_series(x): if lognormal: rands = np.asanyarray( [ - prng.lognormal(mean=mu[i], sigma=sigma[i]) + prng.lognormal(mean=mu[i], sigma=sigma[i], size=size) for i, prng in enumerate(generators) ] ) else: rands = np.asanyarray( [ - prng.normal(loc=mu[i], scale=sigma[i]) + prng.normal(loc=mu[i], scale=sigma[i], size=size) for i, prng in enumerate(generators) ] ) # update offset for rows we handled - self.row_states.loc[df.index, "offset"] += 1 + if size is not None: + consume_offsets = int(size) + else: + consume_offsets = 1 + self.row_states.loc[df.index, "offset"] += consume_offsets return rands @@ -612,7 +616,7 @@ def random_for_df(self, df, n=1): rands = channel.random_for_df(df, self.step_name, n) return rands - def normal_for_df(self, df, mu=0, sigma=1, broadcast=False): + def normal_for_df(self, df, mu=0, sigma=1, broadcast=False, size=None): """ Return a single floating point normal random number in range (-inf, inf) for each row in df using the appropriate random channel for each row. @@ -651,13 +655,16 @@ def normal_for_df(self, df, mu=0, sigma=1, broadcast=False): alts_df = df df = df.index.unique().to_series() rands = channel.normal_for_df( - df, self.step_name, mu=0, sigma=1, lognormal=False + df, self.step_name, mu=0, sigma=1, lognormal=False, size=size ) - rands = reindex(pd.Series(rands, index=df.index), alts_df.index) + if size is not None: + rands = reindex(pd.DataFrame(rands, index=df.index), alts_df.index) + else: + rands = reindex(pd.Series(rands, index=df.index), alts_df.index) rands = rands * sigma + mu else: rands = channel.normal_for_df( - df, self.step_name, mu, sigma, lognormal=False + df, self.step_name, mu, sigma, lognormal=False, size=size ) return rands diff --git a/activitysim/core/simulate.py b/activitysim/core/simulate.py index 01ff427090..5aa681f3ef 100644 --- a/activitysim/core/simulate.py +++ b/activitysim/core/simulate.py @@ -2,23 +2,25 @@ # See full license in LICENSE.txt. import logging +import time import warnings from builtins import range from collections import OrderedDict +from datetime import timedelta import numpy as np import pandas as pd from . import assign, chunk, config, logit, pathbuilder, pipeline, tracing, util +from .simulate_consts import ( + ALT_LOSER_UTIL, + SPEC_DESCRIPTION_NAME, + SPEC_EXPRESSION_NAME, + SPEC_LABEL_NAME, +) logger = logging.getLogger(__name__) -SPEC_DESCRIPTION_NAME = "Description" -SPEC_EXPRESSION_NAME = "Expression" -SPEC_LABEL_NAME = "Label" - -ALT_LOSER_UTIL = -900 - def random_rows(df, n): @@ -319,7 +321,7 @@ def get_segment_coefficients(model_settings, segment_name): FutureWarning, ) else: - raise RuntimeError(f"No COEFFICIENTS setting in model_settings") + raise RuntimeError("No COEFFICIENTS setting in model_settings") if legacy: constants = config.get_model_constants(model_settings) @@ -397,6 +399,11 @@ def eval_coefficients(spec, coefficients, estimator): spec[c].apply(lambda x: eval(str(x), {}, coefficients)).astype(np.float32) ) + sharrow_enabled = config.setting("sharrow", False) + if sharrow_enabled: + # keep all zero rows, reduces the number of unique flows to compile and store. + return spec + # drop any rows with all zeros since they won't have any effect (0 marginal utility) # (do not drop rows in estimation mode as it may confuse the estimation package (e.g. larch) zero_rows = (spec == 0).all(axis=1) @@ -420,8 +427,11 @@ def eval_utilities( estimator=None, trace_column_names=None, log_alt_losers=False, + zone_layer=None, + spec_sh=None, ): """ + Evaluate a utility function as defined in a spec file. Parameters ---------- @@ -432,99 +442,163 @@ def eval_utilities( choosers : pandas.DataFrame locals_d : Dict or None This is a dictionary of local variables that will be the environment - for an evaluation of an expression that begins with @ - trace_label: str - have_trace_targets: boolean - choosers has targets to trace - trace_all_rows: boolean - trace all chooser rows, bypassing tracing.trace_targets + for an evaluation of an expression that begins with "@". + trace_label : str + have_trace_targets : bool + Indicates if `choosers` has targets to trace + trace_all_rows : bool + Trace all chooser rows, bypassing tracing.trace_targets estimator : called to report intermediate table results (used for estimation) - trace_column_names: str or list of str + trace_column_names: str or list[str] chooser columns to include when tracing expression_values + log_alt_losers : bool, default False + Write out expressions when all alternatives are unavailable. + This can be useful for model development to catch errors in + specifications. Enabling this check does not alter valid results + but slows down model runs. + zone_layer : {'taz', 'maz'}, optional + Specify which zone layer of the skims is to be used by sharrow. You + cannot use the 'maz' zone layer in a one-zone model, but you can use + the 'taz' layer in a two- or three-zone model (e.g. for destination + pre-sampling). If not given, the default (lowest available) layer is + used. + spec_sh : pandas.DataFrame, optional + An alternative `spec` modified specifically for use with sharrow. + This is meant to give the same result, but allows for some optimizations + or preprocessing outside the sharrow framework (e.g. to run the Python + based transit virtual path builder and cache relevant values). Returns ------- - + utilities : pandas.DataFrame """ + start_time = time.time() - # fixme - restore tracing and _check_for_variability + sharrow_enabled = config.setting("sharrow", False) - trace_label = tracing.extend_trace_label(trace_label, "eval_utils") + expression_values = None - # avoid altering caller's passed-in locals_d parameter (they may be looping) - locals_dict = assign.local_utilities() + from .flow import TimeLogger - if locals_d is not None: - locals_dict.update(locals_d) - globals_dict = {} + timelogger = TimeLogger("simulate") + sh_util = None + sh_flow = None + utilities = None - locals_dict["df"] = choosers + if spec_sh is None: + spec_sh = spec - # - eval spec expressions - if isinstance(spec.index, pd.MultiIndex): - # spec MultiIndex with expression and label - exprs = spec.index.get_level_values(SPEC_EXPRESSION_NAME) - else: - exprs = spec.index + if locals_d is not None and "disable_sharrow" in locals_d: + sharrow_enabled = False + + if sharrow_enabled: + from .flow import apply_flow # import inside func to prevent circular imports - expression_values = np.empty((spec.shape[0], choosers.shape[0])) - chunk.log_df(trace_label, "expression_values", expression_values) + locals_dict = {} + locals_dict.update(config.get_global_constants()) + if locals_d is not None: + locals_dict.update(locals_d) + sh_util, sh_flow = apply_flow( + spec_sh, + choosers, + locals_dict, + trace_label, + sharrow_enabled == "require", + zone_layer=zone_layer, + ) + utilities = sh_util + timelogger.mark("sharrow flow", True, logger, trace_label) + else: + timelogger.mark("sharrow flow", False) - i = 0 - for expr, coefficients in zip(exprs, spec.values): + # fixme - restore tracing and _check_for_variability - try: - with warnings.catch_warnings(record=True) as w: - # Cause all warnings to always be triggered. - warnings.simplefilter("always") - if expr.startswith("@"): - expression_value = eval(expr[1:], globals_dict, locals_dict) + if utilities is None or estimator or sharrow_enabled == "test": - else: - expression_value = choosers.eval(expr) + trace_label = tracing.extend_trace_label(trace_label, "eval_utils") - if len(w) > 0: - for wrn in w: - logger.warning( - f"{trace_label} - {type(wrn).__name__} ({wrn.message}) evaluating: {str(expr)}" - ) + # avoid altering caller's passed-in locals_d parameter (they may be looping) + locals_dict = assign.local_utilities() - except Exception as err: - logger.exception( - f"{trace_label} - {type(err).__name__} ({str(err)}) evaluating: {str(expr)}" - ) - raise err + if locals_d is not None: + locals_dict.update(locals_d) + globals_dict = {} - if log_alt_losers: - # utils for each alt for this expression - # FIXME if we always did tis, we cold uem these and skip np.dot below - utils = np.outer(expression_value, coefficients) - losers = np.amax(utils, axis=1) < ALT_LOSER_UTIL + locals_dict["df"] = choosers - if losers.any(): - logger.warning( - f"{trace_label} - {sum(losers)} choosers of {len(losers)} " - f"with prohibitive utilities for all alternatives for expression: {expr}" + # - eval spec expressions + if isinstance(spec.index, pd.MultiIndex): + # spec MultiIndex with expression and label + exprs = spec.index.get_level_values(SPEC_EXPRESSION_NAME) + else: + exprs = spec.index + + expression_values = np.empty((spec.shape[0], choosers.shape[0])) + chunk.log_df(trace_label, "expression_values", expression_values) + + i = 0 + for expr, coefficients in zip(exprs, spec.values): + + try: + with warnings.catch_warnings(record=True) as w: + # Cause all warnings to always be triggered. + warnings.simplefilter("always") + if expr.startswith("@"): + expression_value = eval(expr[1:], globals_dict, locals_dict) + else: + expression_value = choosers.eval(expr) + + if len(w) > 0: + for wrn in w: + logger.warning( + f"{trace_label} - {type(wrn).__name__} ({wrn.message}) evaluating: {str(expr)}" + ) + + except Exception as err: + logger.exception( + f"{trace_label} - {type(err).__name__} ({str(err)}) evaluating: {str(expr)}" ) + raise err + + if log_alt_losers: + # utils for each alt for this expression + # FIXME if we always did tis, we cold uem these and skip np.dot below + utils = np.outer(expression_value, coefficients) + losers = np.amax(utils, axis=1) < ALT_LOSER_UTIL - expression_values[i] = expression_value - i += 1 + if losers.any(): + logger.warning( + f"{trace_label} - {sum(losers)} choosers of {len(losers)} " + f"with prohibitive utilities for all alternatives for expression: {expr}" + ) - chunk.log_df(trace_label, "expression_values", expression_values) + expression_values[i] = expression_value + i += 1 - if estimator: - df = pd.DataFrame( - data=expression_values.transpose(), - index=choosers.index, - columns=spec.index.get_level_values(SPEC_LABEL_NAME), + chunk.log_df(trace_label, "expression_values", expression_values) + + if estimator: + df = pd.DataFrame( + data=expression_values.transpose(), + index=choosers.index, + columns=spec.index.get_level_values(SPEC_LABEL_NAME), + ) + df.index.name = choosers.index.name + estimator.write_expression_values(df) + + # - compute_utilities + utilities = np.dot( + expression_values.transpose(), spec.astype(np.float64).values ) - df.index.name = choosers.index.name - estimator.write_expression_values(df) - # - compute_utilities - utilities = np.dot(expression_values.transpose(), spec.astype(np.float64).values) - utilities = pd.DataFrame(data=utilities, index=choosers.index, columns=spec.columns) + timelogger.mark("simple flow", True, logger=logger, suffix=trace_label) + else: + timelogger.mark("simple flow", False) + utilities = pd.DataFrame(data=utilities, index=choosers.index, columns=spec.columns) chunk.log_df(trace_label, "utilities", utilities) + timelogger.mark("assemble utilities") # sometimes tvpb will drop rows on the fly and we wind up with an empty # table of choosers. this will just bypass tracing in that case. @@ -539,39 +613,66 @@ def eval_utilities( # get int offsets of the trace_targets (offsets of bool=True values) offsets = np.nonzero(list(trace_targets))[0] + # trace sharrow + if sh_flow is not None: + try: + data_sh = sh_flow.load( + sh_flow.tree.replace_datasets( + df=choosers.iloc[offsets], + ), + dtype=np.float32, + ) + expression_values_sh = pd.DataFrame(data=data_sh.T, index=spec.index) + except ValueError: + expression_values_sh = None + else: + expression_values_sh = None + # get array of expression_values # expression_values.shape = (len(spec), len(choosers)) # data.shape = (len(spec), len(offsets)) - data = expression_values[:, offsets] + if expression_values is not None: + data = expression_values[:, offsets] - # index is utility expressions (and optional label if MultiIndex) - expression_values_df = pd.DataFrame(data=data, index=spec.index) + # index is utility expressions (and optional label if MultiIndex) + expression_values_df = pd.DataFrame(data=data, index=spec.index) - if trace_column_names is not None: - if isinstance(trace_column_names, str): - trace_column_names = [trace_column_names] - expression_values_df.columns = pd.MultiIndex.from_frame( - choosers.loc[trace_targets, trace_column_names] - ) + if trace_column_names is not None: + if isinstance(trace_column_names, str): + trace_column_names = [trace_column_names] + expression_values_df.columns = pd.MultiIndex.from_frame( + choosers.loc[trace_targets, trace_column_names] + ) + else: + expression_values_df = None - tracing.trace_df( - expression_values_df, - tracing.extend_trace_label(trace_label, "expression_values"), - slicer=None, - transpose=False, - ) + if expression_values_sh is not None: + tracing.trace_df( + expression_values_sh, + tracing.extend_trace_label(trace_label, "expression_values_sh"), + slicer=None, + transpose=False, + ) + if expression_values_df is not None: + tracing.trace_df( + expression_values_df, + tracing.extend_trace_label(trace_label, "expression_values"), + slicer=None, + transpose=False, + ) - if len(spec.columns) > 1: + if len(spec.columns) > 1: - for c in spec.columns: - name = f"expression_value_{c}" + for c in spec.columns: + name = f"expression_value_{c}" - tracing.trace_df( - expression_values_df.multiply(spec[c].values, axis=0), - tracing.extend_trace_label(trace_label, name), - slicer=None, - transpose=False, - ) + tracing.trace_df( + expression_values_df.multiply(spec[c].values, axis=0), + tracing.extend_trace_label(trace_label, name), + slicer=None, + transpose=False, + ) + timelogger.mark("trace", True, logger, trace_label) del expression_values chunk.log_df(trace_label, "expression_values", None) @@ -579,6 +680,43 @@ def eval_utilities( # no longer our problem - but our caller should re-log this... chunk.log_df(trace_label, "utilities", None) + if sharrow_enabled == "test": + try: + np.testing.assert_allclose( + sh_util, + utilities.values, + rtol=1e-2, + atol=0, + err_msg="utility not aligned", + verbose=True, + ) + except AssertionError as err: + print(err) + misses = np.where(~np.isclose(sh_util, utilities.values, rtol=1e-2, atol=0)) + _sh_util_miss1 = sh_util[tuple(m[0] for m in misses)] + _u_miss1 = utilities.values[tuple(m[0] for m in misses)] + diff = _sh_util_miss1 - _u_miss1 + if len(misses[0]) > sh_util.size * 0.01: + print( + f"big problem: {len(misses[0])} missed close values " + f"out of {sh_util.size} ({100*len(misses[0]) / sh_util.size:.2f}%)" + ) + print(f"{sh_util.shape=}") + print(misses) + raise + except TypeError as err: + print(err) + print("sh_util") + print(sh_util) + print("utilities") + print(utilities) + timelogger.mark("sharrow test", True, logger, trace_label) + + end_time = time.time() + logger.info( + f"simulate.eval_utils runtime: {timedelta(seconds=end_time - start_time)} {trace_label}" + ) + timelogger.summary(logger, "simulate.eval_utils timing") return utilities @@ -1070,8 +1208,10 @@ def eval_nl( if have_trace_targets: tracing.trace_df(choosers, "%s.choosers" % trace_label) + choosers, spec_sh = _preprocess_tvpb_logsums_on_choosers(choosers, spec, locals_d) + raw_utilities = eval_utilities( - spec, + spec_sh, choosers, locals_d, log_alt_losers=log_alt_losers, @@ -1079,6 +1219,7 @@ def eval_nl( have_trace_targets=have_trace_targets, estimator=estimator, trace_column_names=trace_column_names, + spec_sh=spec_sh, ) chunk.log_df(trace_label, "raw_utilities", raw_utilities) @@ -1345,7 +1486,7 @@ def simple_simulate( result_list.append(choices) - chunk.log_df(trace_label, f"result_list", result_list) + chunk.log_df(trace_label, "result_list", result_list) if len(result_list) > 1: choices = pd.concat(result_list) @@ -1396,7 +1537,7 @@ def simple_simulate_by_chunk_id( result_list.append(choices) - chunk.log_df(trace_label, f"result_list", result_list) + chunk.log_df(trace_label, "result_list", result_list) if len(result_list) > 1: choices = pd.concat(result_list) @@ -1452,6 +1593,85 @@ def eval_mnl_logsums(choosers, spec, locals_d, trace_label=None): return logsums +def _preprocess_tvpb_logsums_on_choosers(choosers, spec, locals_d): + """ + Compute TVPB logsums and attach those values to the choosers. + + Also generate a modified spec that uses the replacement value instead of + regenerating the logsums dynamically inline. + + Parameters + ---------- + choosers + spec + locals_d + + Returns + ------- + choosers + spec + + """ + spec_sh = spec.copy() + + def _replace_in_level(multiindex, level_name, *args, **kwargs): + y = multiindex.levels[multiindex.names.index(level_name)].str.replace( + *args, **kwargs + ) + return multiindex.set_levels(y, level=level_name) + + # Preprocess TVPB logsums outside sharrow + if "tvpb_logsum_odt" in locals_d: + tvpb = locals_d["tvpb_logsum_odt"] + path_types = tvpb.tvpb.network_los.setting( + f"TVPB_SETTINGS.{tvpb.recipe}.path_types" + ).keys() + assignments = {} + for path_type in ["WTW", "DTW"]: + if path_type not in path_types: + continue + re_spec = spec_sh.index + re_spec = _replace_in_level( + re_spec, + "Expression", + rf"tvpb_logsum_odt\['{path_type}'\]", + f"df.PRELOAD_tvpb_logsum_odt_{path_type}", + regex=True, + ) + if not all(spec_sh.index == re_spec): + spec_sh.index = re_spec + preloaded = locals_d["tvpb_logsum_odt"][path_type] + assignments[f"PRELOAD_tvpb_logsum_odt_{path_type}"] = preloaded + if assignments: + choosers = choosers.assign(**assignments) + + if "tvpb_logsum_dot" in locals_d: + tvpb = locals_d["tvpb_logsum_dot"] + path_types = tvpb.tvpb.network_los.setting( + f"TVPB_SETTINGS.{tvpb.recipe}.path_types" + ).keys() + assignments = {} + for path_type in ["WTW", "WTD"]: + if path_type not in path_types: + continue + re_spec = spec_sh.index + re_spec = _replace_in_level( + re_spec, + "Expression", + rf"tvpb_logsum_dot\['{path_type}'\]", + f"df.PRELOAD_tvpb_logsum_dot_{path_type}", + regex=True, + ) + if not all(spec_sh.index == re_spec): + spec_sh.index = re_spec + preloaded = locals_d["tvpb_logsum_dot"][path_type] + assignments[f"PRELOAD_tvpb_logsum_dot_{path_type}"] = preloaded + if assignments: + choosers = choosers.assign(**assignments) + + return choosers, spec_sh + + def eval_nl_logsums(choosers, spec, nest_spec, locals_d, trace_label=None): """ like eval_nl except return logsums instead of making choices @@ -1467,16 +1687,19 @@ def eval_nl_logsums(choosers, spec, nest_spec, locals_d, trace_label=None): logit.validate_nest_spec(nest_spec, trace_label) + choosers, spec_sh = _preprocess_tvpb_logsums_on_choosers(choosers, spec, locals_d) + # trace choosers if have_trace_targets: tracing.trace_df(choosers, "%s.choosers" % trace_label) raw_utilities = eval_utilities( - spec, + spec_sh, choosers, locals_d, trace_label=trace_label, have_trace_targets=have_trace_targets, + spec_sh=spec_sh, ) chunk.log_df(trace_label, "raw_utilities", raw_utilities) @@ -1576,7 +1799,7 @@ def simple_simulate_logsums( result_list.append(logsums) - chunk.log_df(trace_label, f"result_list", result_list) + chunk.log_df(trace_label, "result_list", result_list) if len(result_list) > 1: logsums = pd.concat(result_list) diff --git a/activitysim/core/simulate_consts.py b/activitysim/core/simulate_consts.py new file mode 100644 index 0000000000..dc36b640d9 --- /dev/null +++ b/activitysim/core/simulate_consts.py @@ -0,0 +1,5 @@ +SPEC_DESCRIPTION_NAME = "Description" +SPEC_EXPRESSION_NAME = "Expression" +SPEC_LABEL_NAME = "Label" + +ALT_LOSER_UTIL = -900 diff --git a/activitysim/core/skim_dataset.py b/activitysim/core/skim_dataset.py new file mode 100644 index 0000000000..24166f46f7 --- /dev/null +++ b/activitysim/core/skim_dataset.py @@ -0,0 +1,774 @@ +import glob +import logging +import os + +import numpy as np +import openmatrix +import pandas as pd +import sharrow as sh + +from . import config +from . import flow as __flow # noqa, keep this here for side effects? +from . import inject + +logger = logging.getLogger(__name__) + +POSITIONS_AS_DICT = True + + +class SkimDataset: + def __init__(self, dataset): + self.dataset = dataset + self.time_map = { + j: i for i, j in enumerate(self.dataset.indexes["time_period"]) + } + self.usage = set() # track keys of skims looked up + + @property + def odim(self): + if "omaz" in self.dataset.dims: + return "omaz" + else: + return "otaz" + + @property + def ddim(self): + if "dmaz" in self.dataset.dims: + return "dmaz" + else: + return "dtaz" + + def get_skim_usage(self): + """ + return set of keys of skims looked up. e.g. {'DIST', 'SOV'} + + Returns + ------- + set: + """ + return self.usage + + def wrap(self, orig_key, dest_key): + """ + return a SkimWrapper for self + """ + return DatasetWrapper(self.dataset, orig_key, dest_key, time_map=self.time_map) + + def wrap_3d(self, orig_key, dest_key, dim3_key): + """ + return a SkimWrapper for self + """ + return DatasetWrapper( + self.dataset, orig_key, dest_key, dim3_key, time_map=self.time_map + ) + + def lookup(self, orig, dest, key): + self.usage.add(key) + use_index = None + + if use_index is None and hasattr(orig, "index"): + use_index = orig.index + if use_index is None and hasattr(dest, "index"): + use_index = dest.index + + orig = np.asanyarray(orig).astype(int) + dest = np.asanyarray(dest).astype(int) + + # TODO offset mapper if required + positions = {self.odim: orig, self.ddim: dest} + + # When asking for a particular time period + if isinstance(key, tuple) and len(key) == 2: + main_key, time_key = key + if time_key in self.time_map: + positions["time_period"] = np.full_like(orig, self.time_map[time_key]) + key = main_key + else: + raise KeyError(key) + + result = self.dataset.iat(**positions, _name=key) # iat strips data encoding + # result = igather(self.dataset[key], positions) + # + # if 'digital_encoding' in self.dataset[key].attrs: + # result = array_decode(result, self.dataset[key].attrs['digital_encoding']) + + result = result.to_series() + + if use_index is not None: + result.index = use_index + return result + + def map_time_periods_from_series(self, time_period_labels): + logger.info(f"vectorize lookup for time_period={time_period_labels.name}") + time_period_idxs = pd.Series( + np.vectorize(self.time_map.get)(time_period_labels), + index=time_period_labels.index, + ) + return time_period_idxs + + +class DatasetWrapper: + def __init__(self, dataset, orig_key, dest_key, time_key=None, *, time_map=None): + """ + + Parameters + ---------- + skim_dict: SkimDict + + orig_key: str + name of column in dataframe to use as implicit orig for lookups + dest_key: str + name of column in dataframe to use as implicit dest for lookups + """ + self.dataset = dataset + self.orig_key = orig_key + self.dest_key = dest_key + self.time_key = time_key + self.df = None + if time_map is None: + self.time_map = { + j: i for i, j in enumerate(self.dataset.indexes["time_period"]) + } + else: + self.time_map = time_map + + @property + def odim(self): + if "omaz" in self.dataset.dims: + return "omaz" + else: + return "otaz" + + @property + def ddim(self): + if "dmaz" in self.dataset.dims: + return "dmaz" + else: + return "dtaz" + + def map_time_periods(self, df): + if self.time_key: + logger.info(f"vectorize lookup for time_period={self.time_key}") + time_period_idxs = pd.Series( + np.vectorize(self.time_map.get)(df[self.time_key]), + index=df.index, + ) + return time_period_idxs + + def set_df(self, df): + """ + Set the dataframe + + Parameters + ---------- + df : DataFrame + The dataframe which contains the origin and destination ids + + Returns + ------- + self (to facilitate chaining) + """ + assert ( + self.orig_key in df + ), f"orig_key '{self.orig_key}' not in df columns: {list(df.columns)}" + assert ( + self.dest_key in df + ), f"dest_key '{self.dest_key}' not in df columns: {list(df.columns)}" + if self.time_key: + assert ( + self.time_key in df + ), f"time_key '{self.time_key}' not in df columns: {list(df.columns)}" + self.df = df + + # TODO allow offsets if needed + positions = { + self.odim: df[self.orig_key], + self.ddim: df[self.dest_key], + } + if self.time_key: + if ( + np.issubdtype(df[self.time_key].dtype, np.integer) + and df[self.time_key].max() < self.dataset.dims["time_period"] + ): + logger.info(f"natural use for time_period={self.time_key}") + positions["time_period"] = df[self.time_key] + else: + logger.info(f"vectorize lookup for time_period={self.time_key}") + positions["time_period"] = pd.Series( + np.vectorize(self.time_map.get)(df[self.time_key]), + index=df.index, + ) + + if POSITIONS_AS_DICT: + self.positions = {} + for k, v in positions.items(): + self.positions[k] = v.astype(int) + else: + self.positions = pd.DataFrame(positions).astype(int) + + return self + + def lookup(self, key, reverse=False): + """ + Generally not called by the user - use __getitem__ instead + + Parameters + ---------- + key : hashable + The key (identifier) for this skim object + + od : bool (optional) + od=True means lookup standard origin-destination skim value + od=False means lookup destination-origin skim value + + Returns + ------- + impedances: pd.Series + A Series of impedances which are elements of the Skim object and + with the same index as df + """ + + assert self.df is not None, "Call set_df first" + if reverse: + if isinstance(self.positions, dict): + x = self.positions.copy() + x.update( + { + self.odim: self.positions[self.ddim], + self.ddim: self.positions[self.odim], + } + ) + else: + x = self.positions.rename( + columns={self.odim: self.ddim, self.ddim: self.odim} + ) + else: + if isinstance(self.positions, dict): + x = self.positions.copy() + else: + x = self.positions + + # When asking for a particular time period + if isinstance(key, tuple) and len(key) == 2: + main_key, time_key = key + if time_key in self.time_map: + if isinstance(x, dict): + x["time_period"] = np.full_like( + x[self.odim], fill_value=self.time_map[time_key] + ) + else: + x = x.assign(time_period=self.time_map[time_key]) + key = main_key + else: + raise KeyError(key) + + result = self.dataset.iat(**x, _name=key) # iat strips data encoding + # if 'digital_encoding' in self.dataset[key].attrs: + # result = array_decode(result, self.dataset[key].attrs['digital_encoding']) + + # Return a series, consistent with ActivitySim SkimWrapper + out = result.to_series() + out.index = self.df.index + return out + + def reverse(self, key): + """ + return skim value in reverse (d-o) direction + """ + return self.lookup(key, reverse=True) + + def max(self, key): + """ + return max skim value in either o-d or d-o direction + """ + assert self.df is not None, "Call set_df first" + + s = np.maximum( + self.lookup(key), + self.lookup(key, True), + ) + + return pd.Series(s, index=self.df.index) + + def __getitem__(self, key): + """ + Get the lookup for an available skim object (df and orig/dest and column names implicit) + + Parameters + ---------- + key : hashable + The key (identifier) for the skim object + + Returns + ------- + impedances: pd.Series with the same index as df + A Series of impedances values from the single Skim with specified key, indexed byt orig/dest pair + """ + return self.lookup(key) + + +def _should_invalidate_cache_file(cache_filename, *source_filenames): + """ + Check if a cache file should be invalidated. + + It should be invalidated if any source file has a modification time + more recent than the cache file modification time. + + Parameters + ---------- + cache_filename : Path-like + source_filenames : Collection[Path-like] + + Returns + ------- + bool + """ + from stat import ST_MTIME + + try: + stat0 = os.stat(cache_filename) + except FileNotFoundError: + # cache file does not even exist + return True + for i in source_filenames: + stat1 = os.stat(i) + if stat0[ST_MTIME] < stat1[ST_MTIME]: + return True + return False + + +def _use_existing_backing_if_valid(backing, omx_file_paths, skim_tag): + """ + Open an xarray dataset from a backing store if possible. + + Parameters + ---------- + backing : str + What kind of memory backing to use. Memmaps always start + with "memmap:" and then have a file system location, so + if this pattern does not apply the backing is not a memmap, + and instead the backing string is used as the key to find + the data in ephemeral shared memory. + omx_file_paths : Collection[Path-like] + These are the original source files. If the file modification + time for any of these files is more recent than the memmap, + the memmap files are invalid and will be deleted so they + can be rebuilt. + skim_tag : str + For error message reporting only + + Returns + ------- + xarray.Dataset or None + """ + out = None + if backing.startswith("memmap:"): + # when working with a memmap, check if the memmap file on disk + # needs to be invalidated, because the source skims have been + # modified more recently. + if not _should_invalidate_cache_file(backing[7:], *omx_file_paths): + try: + out = sh.Dataset.shm.from_shared_memory(backing, mode="r") + except FileNotFoundError as err: + logger.info(f"skim dataset {skim_tag!r} not found {err!s}") + logger.info(f"loading skim dataset {skim_tag!r} from original sources") + out = None + else: + logger.info("using skim_dataset from shared memory") + else: + sh.Dataset.shm.delete_shared_memory_files(backing) + else: + # when working in ephemeral shared memory, assume that if that data + # is loaded then it is good to use without further checks. + try: + out = sh.Dataset.shm.from_shared_memory(backing, mode="r") + except FileNotFoundError as err: + logger.info(f"skim dataset {skim_tag!r} not found {err!s}") + logger.info(f"loading skim dataset {skim_tag!r} from original sources") + out = None + return out + + +def _dedupe_time_periods(network_los_preload): + raw_time_periods = network_los_preload.los_settings["skim_time_periods"]["labels"] + # deduplicate time period names + time_periods = [] + for t in raw_time_periods: + if t not in time_periods: + time_periods.append(t) + return time_periods + + +def _apply_digital_encoding(dataset, digital_encodings): + """ + Apply digital encoding to compress skims with minimal information loss. + + Parameters + ---------- + dataset : xarray.Dataset + digital_encodings : Collection[Dict] + A collection of digital encoding instructions. To apply the same + encoding for multiple variables, the Dict should have a 'regex' key + that gives a regular expressing to match. Otherwise, see + sharrow's digital_encoding for other details. + + Returns + ------- + dataset : xarray.Dataset + As modified + """ + if digital_encodings: + import re + + # apply once, before saving to zarr, will stick around in cache + for encoding in digital_encodings: + logger.info(f"applying zarr digital-encoding: {encoding}") + regex = encoding.pop("regex", None) + joint_dict = encoding.pop("joint_dict", None) + if joint_dict: + joins = [] + for k in dataset.variables: + assert isinstance(k, str) # variable names should be strings + if re.match(regex, k): + joins.append(k) + dataset = dataset.digital_encoding.set( + joins, joint_dict=joint_dict, **encoding + ) + elif regex: + if "name" in encoding: + raise ValueError( + "cannot give both name and regex for digital_encoding" + ) + for k in dataset.variables: + assert isinstance(k, str) # variable names should be strings + if re.match(regex, k): + dataset = dataset.digital_encoding.set(k, **encoding) + else: + dataset = dataset.digital_encoding.set(**encoding) + return dataset + + +def _scan_for_unused_names(tokens): + """ + Scan all spec files to find unused skim variable names. + + Parameters + ---------- + tokens : Collection[str] + + Returns + ------- + Set[str] + """ + configs_dir_list = inject.get_injectable("configs_dir") + configs_dir_list = ( + [configs_dir_list] if isinstance(configs_dir_list, str) else configs_dir_list + ) + assert isinstance(configs_dir_list, list) + + for directory in configs_dir_list: + logger.debug(f"scanning for unused skims in {directory}") + filenames = glob.glob(os.path.join(directory, "*.csv")) + for filename in filenames: + with open(filename, "rt") as f: + content = f.read() + missing_tokens = set() + for t in tokens: + if t not in content: + missing_tokens.add(t) + tokens = missing_tokens + if not tokens: + return tokens + return tokens + + +def _drop_unused_names(dataset): + logger.info("scanning for unused skims") + tokens = set(dataset.variables.keys()) - set(dataset.coords.keys()) + unused_tokens = _scan_for_unused_names(tokens) + if unused_tokens: + baggage = dataset.digital_encoding.baggage(None) + unused_tokens -= baggage + # retain sparse matrix tables + unused_tokens = set(i for i in unused_tokens if not i.startswith("_s_")) + # retain lookup tables + unused_tokens = set(i for i in unused_tokens if not i.startswith("_digitized_")) + logger.info(f"dropping unused skims: {unused_tokens}") + dataset = dataset.drop_vars(unused_tokens) + else: + logger.info("no unused skims found") + return dataset + + +def load_sparse_maz_skims( + dataset, + land_use_index, + remapper, + zone_system, + maz2taz_file_name, + maz_to_maz_tables=(), + max_blend_distance=None, + data_file_resolver=None, +): + """ + Load sparse MAZ data on top of TAZ skim data. + + Parameters + ---------- + dataset : xarray.Dataset + The existing dataset at TAZ resolution only. + land_use_index : pandas.Index + The index of the land use table. For two and three zone systems, + these index values should be MAZ identifiers. + remapper : dict, optional + A dictionary mapping where the keys are the original (nominal) zone + id's, and the values are the recoded (typically zero-based contiguous) + zone id's. Recoding improves runtime efficiency. + zone_system : int + Currently 1, 2 and 3 are supported. + maz2taz_file_name : str + maz_to_maz_tables : Collection[] + max_blend_distance : optional + data_file_resolver : function + + Returns + ------- + xarray.Dataset + """ + from ..core.los import THREE_ZONE, TWO_ZONE + + if data_file_resolver is None: + data_file_resolver = config.data_file_path + + if zone_system in [TWO_ZONE, THREE_ZONE]: + + # maz + maz_taz = pd.read_csv(data_file_resolver(maz2taz_file_name, mandatory=True)) + maz_taz = maz_taz[["MAZ", "TAZ"]].set_index("MAZ").sort_index() + + # MAZ alignment is ensured here, so no re-alignment check is + # needed below for TWO_ZONE or THREE_ZONE systems + try: + pd.testing.assert_index_equal( + maz_taz.index, land_use_index, check_names=False + ) + except AssertionError: + if remapper is not None: + maz_taz.index = maz_taz.index.map(remapper.get) + maz_taz = maz_taz.sort_index() + assert maz_taz.index.equals( + land_use_index.sort_values() + ), "maz-taz lookup index does not match index of land_use table" + else: + raise + + dataset.redirection.set( + maz_taz, + map_to="otaz", + name="omaz", + map_also={"dtaz": "dmaz"}, + ) + + maz_to_maz_tables = ( + [maz_to_maz_tables] + if isinstance(maz_to_maz_tables, str) + else maz_to_maz_tables + ) + + if max_blend_distance is None: + max_blend_distance = {} + if isinstance(max_blend_distance, int): + max_blend_distance = {"DEFAULT": max_blend_distance} + + for file_name in maz_to_maz_tables: + + df = pd.read_csv(data_file_resolver(file_name, mandatory=True)) + if remapper is not None: + df.OMAZ = df.OMAZ.map(remapper.get) + df.DMAZ = df.DMAZ.map(remapper.get) + for colname in df.columns: + if colname in ["OMAZ", "DMAZ"]: + continue + max_blend_distance_i = max_blend_distance.get("DEFAULT", None) + max_blend_distance_i = max_blend_distance.get( + colname, max_blend_distance_i + ) + dataset.redirection.sparse_blender( + colname, + df.OMAZ, + df.DMAZ, + df[colname], + max_blend_distance=max_blend_distance_i, + index=land_use_index, + ) + + return dataset + + +def _skim_dataset(skim_tag="taz"): + from ..core.los import ONE_ZONE + + # TODO:SHARROW: taz and maz are the same + network_los_preload = inject.get_injectable("network_los_preload", None) + if network_los_preload is None: + raise ValueError("missing network_los_preload") + + # find which OMX files are to be used. + omx_file_paths = config.expand_input_file_list( + network_los_preload.omx_file_names(skim_tag), + ) + zarr_file = network_los_preload.zarr_file_name(skim_tag) + + if config.setting("disable_zarr", False): + # we can disable the zarr optimizations by setting the `disable_zarr` + # flag in the master config file to True + zarr_file = None + + if zarr_file is not None: + zarr_file = os.path.join(config.get_cache_dir(), zarr_file) + + max_float_precision = network_los_preload.skim_max_float_precision(skim_tag) + + skim_digital_encoding = network_los_preload.skim_digital_encoding(skim_tag) + zarr_digital_encoding = network_los_preload.zarr_pre_encoding(skim_tag) + + # The backing can be plain shared_memory, or a memmap + backing = network_los_preload.skim_backing_store(skim_tag) + if backing == "memmap": + # if memmap is given without a path, create a cache file + mmap_file = os.path.join( + config.get_cache_dir(), f"sharrow_dataset_{skim_tag}.mmap" + ) + backing = f"memmap:{mmap_file}" + + land_use = inject.get_table("land_use") + + if f"_original_{land_use.index.name}" in land_use.to_frame(): + land_use_zone_ids = land_use.to_frame()[f"_original_{land_use.index.name}"] + remapper = dict(zip(land_use_zone_ids, land_use_zone_ids.index)) + else: + remapper = None + + d = _use_existing_backing_if_valid(backing, omx_file_paths, skim_tag) + + if d is None: + time_periods = _dedupe_time_periods(network_los_preload) + if zarr_file: + logger.info(f"looking for zarr skims at {zarr_file}") + if zarr_file and os.path.exists(zarr_file): + # TODO: check if the OMX skims or sparse MAZ are modified more + # recently than the cached ZARR versions; if so do not use + # the ZARR + logger.info("found zarr skims, loading them") + d = sh.dataset.from_zarr_with_attr(zarr_file).max_float_precision( + max_float_precision + ) + else: + if zarr_file: + logger.info("did not find zarr skims, loading omx") + d = sh.dataset.from_omx_3d( + [openmatrix.open_file(f, mode="r") for f in omx_file_paths], + index_names=( + ("otap", "dtap", "time_period") + if skim_tag == "tap" + else ("otaz", "dtaz", "time_period") + ), + time_periods=time_periods, + max_float_precision=max_float_precision, + ) + + if zarr_file: + try: + import zarr # noqa + + # ensure zarr is available before we do all this work + except ModuleNotFoundError: + logger.warning( + "the 'zarr' package is not installed, " + "cannot cache skims to zarr" + ) + else: + if zarr_digital_encoding: + d = _apply_digital_encoding(d, zarr_digital_encoding) + logger.info(f"writing zarr skims to {zarr_file}") + d.to_zarr_with_attr(zarr_file) + + if skim_tag in ("taz", "maz"): + # load sparse MAZ skims, if any + # these are processed after the ZARR stuff as the GCXS sparse array + # is not yet compatible with ZARR directly. + # see https://github.com/pydata/sparse/issues/222 + # or https://github.com/zarr-developers/zarr-python/issues/424 + maz2taz_file_name = network_los_preload.setting("maz", None) + if maz2taz_file_name: + d = load_sparse_maz_skims( + d, + land_use.index, + remapper, + zone_system=network_los_preload.zone_system, + maz2taz_file_name=network_los_preload.setting("maz"), + maz_to_maz_tables=network_los_preload.setting("maz_to_maz.tables"), + max_blend_distance=network_los_preload.setting( + "maz_to_maz.max_blend_distance", default={} + ), + ) + + d = _drop_unused_names(d) + # apply non-zarr dependent digital encoding + d = _apply_digital_encoding(d, skim_digital_encoding) + + if skim_tag in ("taz", "maz"): + # check alignment of TAZs that it matches land_use table + logger.info("checking skims alignment with land_use") + try: + land_use_zone_id = land_use[f"_original_{land_use.index.name}"] + except KeyError: + land_use_zone_id = land_use.index + else: + land_use_zone_id = None + + if network_los_preload.zone_system == ONE_ZONE: + # check TAZ alignment for ONE_ZONE system. + # other systems use MAZ for most lookups, which dynamically + # resolves to TAZ inside the Dataset code. + if d["otaz"].attrs.get("preprocessed") != "zero-based-contiguous": + try: + np.testing.assert_array_equal(land_use_zone_id, d.otaz) + except AssertionError as err: + logger.info(f"otaz realignment required\n{err}") + d = d.reindex(otaz=land_use_zone_id) + else: + logger.info("otaz alignment ok") + d["otaz"] = land_use.index.to_numpy() + d["otaz"].attrs["preprocessed"] = "zero-based-contiguous" + else: + np.testing.assert_array_equal(land_use.index, d.otaz) + + if d["dtaz"].attrs.get("preprocessed") != "zero-based-contiguous": + try: + np.testing.assert_array_equal(land_use_zone_id, d.dtaz) + except AssertionError as err: + logger.info(f"dtaz realignment required\n{err}") + d = d.reindex(dtaz=land_use_zone_id) + else: + logger.info("dtaz alignment ok") + d["dtaz"] = land_use.index.to_numpy() + d["dtaz"].attrs["preprocessed"] = "zero-based-contiguous" + else: + np.testing.assert_array_equal(land_use.index, d.dtaz) + + if d.shm.is_shared_memory: + return d + else: + logger.info("writing skims to shared memory") + return d.shm.to_shared_memory(backing, mode="r") + + +@inject.injectable(cache=True) +def skim_dataset(): + return _skim_dataset() + + +@inject.injectable(cache=True) +def tap_dataset(): + return _skim_dataset("tap") diff --git a/activitysim/core/skim_dict_factory.py b/activitysim/core/skim_dict_factory.py index 6c1c8f2228..a40409fe3a 100644 --- a/activitysim/core/skim_dict_factory.py +++ b/activitysim/core/skim_dict_factory.py @@ -5,12 +5,12 @@ import logging import multiprocessing import os -from abc import ABC, abstractmethod +from abc import ABC import numpy as np import openmatrix as omx -from activitysim.core import config, inject, skim_dictionary, tracing, util +from activitysim.core import config, inject, skim_dictionary, util logger = logging.getLogger(__name__) @@ -87,7 +87,8 @@ def __init__(self, skim_tag, network_los): self.base_keys = None self.block_offsets = None - self.load_skim_info(skim_tag) + if skim_tag: + self.load_skim_info(skim_tag) def load_skim_info(self, skim_tag): """ @@ -114,7 +115,7 @@ def load_skim_info(self, skim_tag): logger.debug(f"load_skim_info {skim_tag} reading {omx_file_path}") - with omx.open_file(omx_file_path) as omx_file: + with omx.open_file(omx_file_path, mode="r") as omx_file: # fixme call to omx_file.shape() failing in windows p3.5 if self.omx_shape is None: @@ -281,7 +282,7 @@ def _read_skims_from_omx(self, skim_info, skim_data): logger.info(f"_read_skims_from_omx {omx_file_path}") # read skims into skim_data - with omx.open_file(omx_file_path) as omx_file: + with omx.open_file(omx_file_path, mode="r") as omx_file: for skim_key, omx_key in omx_keys.items(): if omx_manifest[omx_key] == omx_file_path: diff --git a/activitysim/core/skim_dictionary.py b/activitysim/core/skim_dictionary.py index d707e560ee..04aa36d0fb 100644 --- a/activitysim/core/skim_dictionary.py +++ b/activitysim/core/skim_dictionary.py @@ -7,8 +7,6 @@ import numpy as np import pandas as pd -from activitysim.core.util import quick_loc_series - logger = logging.getLogger(__name__) NOT_IN_SKIM_ZONE_ID = -1 @@ -126,9 +124,6 @@ def map(self, zone_ids): assert self.offset_int is None assert isinstance(self.offset_series, pd.Series) - # FIXME - turns out it is faster to use series.map if zone_ids is a series - # offsets = quick_loc_series(zone_ids, self.offset_series).fillna(NOT_IN_SKIM_ZONE_ID).astype(int) - if isinstance(zone_ids, np.ndarray): zone_ids = pd.Series(zone_ids) offsets = ( @@ -279,9 +274,12 @@ def _lookup(self, orig, dest, block_offsets): # print(f"in_skim\n{in_skim}") # check for bad indexes (other than NOT_IN_SKIM_ZONE_ID) - assert ( + if not ( in_skim | (orig == NOT_IN_SKIM_ZONE_ID) | (dest == NOT_IN_SKIM_ZONE_ID) - ).all(), f"{(~in_skim).sum()} od pairs not in skim" + ).all(): + raise AssertionError( + f"{(~in_skim).sum()} od pairs not in skim including [{orig[~in_skim][:5]}]->[{dest[~in_skim][:5]}]" + ) if not in_skim.all(): result = np.where(in_skim, result, NOT_IN_SKIM_NAN).astype(self.dtype) @@ -634,7 +632,33 @@ def __init__(self, skim_tag, network_los, taz_skim_dict): self.network_los = network_los - super().__init__(skim_tag, taz_skim_dict.skim_info, taz_skim_dict.skim_data) + from activitysim.core.cleaning import ( + recode_based_on_table, + should_recode_based_on_table, + ) + + if should_recode_based_on_table("land_use_taz"): + from .skim_dict_factory import SkimInfo + + skim_info = SkimInfo(None, network_los) + skim_info.skim_tag = taz_skim_dict.skim_info.skim_tag + skim_info.dtype_name = network_los.skim_dtype_name + skim_info.omx_manifest = taz_skim_dict.skim_info.omx_manifest + skim_info.omx_shape = taz_skim_dict.skim_info.omx_shape + skim_info.num_skims = taz_skim_dict.skim_info.num_skims + skim_info.skim_data_shape = taz_skim_dict.skim_info.skim_data_shape + skim_info.offset_map_name = taz_skim_dict.skim_info.offset_map_name + skim_info.omx_keys = taz_skim_dict.skim_info.omx_keys + skim_info.base_keys = taz_skim_dict.skim_info.base_keys + skim_info.block_offsets = taz_skim_dict.skim_info.block_offsets + + skim_info.offset_map = recode_based_on_table( + taz_skim_dict.skim_info.offset_map, "land_use_taz" + ) + else: + skim_info = taz_skim_dict.skim_info + + super().__init__(skim_tag, skim_info, taz_skim_dict.skim_data) assert ( self.offset_mapper is not None ) # should have been set with _init_offset_mapper @@ -658,20 +682,15 @@ def _offset_mapper(self): OffsetMapper """ - # start with a series with MAZ zone_id index and TAZ zone id values - maz_to_taz = ( - self.network_los.maz_taz_df[["MAZ", "TAZ"]] - .set_index("MAZ") - .sort_values(by="TAZ") - .TAZ - ) - # use taz offset_mapper to create series mapping directly from MAZ to TAZ skim index taz_offset_mapper = super()._offset_mapper() - maz_to_skim_offset = taz_offset_mapper.map(maz_to_taz) + maz_taz = self.network_los.get_maz_to_taz_series + maz_to_skim_offset = taz_offset_mapper.map(maz_taz) if isinstance(maz_to_skim_offset, np.ndarray): - maz_to_skim_offset = pd.Series(maz_to_skim_offset, maz_to_taz.index) # bug + maz_to_skim_offset = pd.Series( + maz_to_skim_offset, self.network_los.get_maz_to_taz_series.index + ) # bug # MAZ # 19062 330 <- The TAZ would be, say, 331, and the offset is 330 @@ -685,6 +704,8 @@ def _offset_mapper(self): offset_mapper = OffsetMapper(offset_series=maz_to_skim_offset) elif isinstance(maz_to_skim_offset, np.ndarray): offset_mapper = OffsetMapper(offset_list=maz_to_skim_offset) + else: + raise NotImplementedError return offset_mapper @@ -877,3 +898,9 @@ def get(self, row_ids, col_ids): result = pd.Series(result, index=row_ids.index) return result + + def get_rows(self, row_ids): + return self.offset_mapper.map(np.asanyarray(row_ids)) + + def get_cols(self, col_ids): + return np.vectorize(self.cols_to_indexes.get)(col_ids) diff --git a/activitysim/core/steps/output.py b/activitysim/core/steps/output.py index 93397b914c..130a6ba356 100644 --- a/activitysim/core/steps/output.py +++ b/activitysim/core/steps/output.py @@ -37,9 +37,16 @@ def track_skim_usage(output_dir): for key in skim_dict.get_skim_usage(): print(key, file=output_file) - unused = set(k for k in skim_dict.skim_info.base_keys) - set( - k for k in skim_dict.get_skim_usage() - ) + try: + unused = set(k for k in skim_dict.skim_info.base_keys) - set( + k for k in skim_dict.get_skim_usage() + ) + except AttributeError: + base_keys = set(skim_dict.dataset.variables.keys()) - set( + skim_dict.dataset.coords.keys() + ) + # using dataset + unused = base_keys - set(k for k in skim_dict.get_skim_usage()) for key in unused: print(key, file=output_file) @@ -269,6 +276,12 @@ def write_tables(output_dir): for table_name in output_tables_list: + if not isinstance(table_name, str): + table_decode_cols = table_name.get("decode_columns", {}) + table_name = table_name["tablename"] + else: + table_decode_cols = {} + if table_name == "checkpoints": df = pipeline.get_checkpoints() else: @@ -304,6 +317,43 @@ def write_tables(output_dir): ) df = df.sort_index() + if config.setting("recode_pipeline_columns", True): + for colname, decode_instruction in table_decode_cols.items(): + if "|" in decode_instruction: + decode_filter, decode_instruction = decode_instruction.split("|") + decode_filter = decode_filter.strip() + decode_instruction = decode_instruction.strip() + else: + decode_filter = None + if "." not in decode_instruction: + lookup_col = decode_instruction + source_table = table_name + parent_table = df + else: + source_table, lookup_col = decode_instruction.split(".") + parent_table = inject.get_table(source_table) + try: + map_col = parent_table[f"_original_{lookup_col}"] + except KeyError: + map_col = parent_table[lookup_col] + map_col = np.asarray(map_col) + map_func = map_col.__getitem__ + if decode_filter: + if decode_filter == "nonnegative": + + def map_func(x): + return x if x < 0 else map_col[x] + + else: + raise ValueError(f"unknown decode_filter {decode_filter}") + if colname in df.columns: + df[colname] = df[colname].astype(int).map(map_func) + elif colname == df.index.name: + df.index = df.index.astype(int).map(map_func) + # drop _original_x from table if it is duplicative + if source_table == table_name and f"_original_{lookup_col}" in df: + df = df.drop(columns=[f"_original_{lookup_col}"]) + if h5_store: file_path = config.output_file_path("%soutput_tables.h5" % prefix) df.to_hdf(file_path, key=table_name, mode="a", format="fixed") diff --git a/activitysim/core/test/test_logit.py b/activitysim/core/test/test_logit.py index 225fdbb3c7..8253149e1f 100644 --- a/activitysim/core/test/test_logit.py +++ b/activitysim/core/test/test_logit.py @@ -116,14 +116,20 @@ def test_make_choices_only_one(): ) choices, rands = logit.make_choices(probs) - pdt.assert_series_equal(choices, pd.Series([0, 1], index=["x", "y"])) + pdt.assert_series_equal( + choices, pd.Series([0, 1], index=["x", "y"]), check_dtype=False + ) def test_make_choices_real_probs(utilities): probs = logit.utils_to_probs(utilities, trace_label=None) choices, rands = logit.make_choices(probs) - pdt.assert_series_equal(choices, pd.Series([1, 2], index=[0, 1])) + pdt.assert_series_equal( + choices, + pd.Series([1, 2], index=[0, 1]), + check_dtype=False, + ) @pytest.fixture(scope="module") diff --git a/activitysim/core/test/test_simulate.py b/activitysim/core/test/test_simulate.py index 9eaa66534a..ab100f6a96 100644 --- a/activitysim/core/test/test_simulate.py +++ b/activitysim/core/test/test_simulate.py @@ -75,7 +75,7 @@ def test_simple_simulate(data, spec): choices = simulate.simple_simulate(choosers=data, spec=spec, nest_spec=None) expected = pd.Series([1, 1, 1], index=data.index) - pdt.assert_series_equal(choices, expected) + pdt.assert_series_equal(choices, expected, check_dtype=False) def test_simple_simulate_chunked(data, spec): @@ -86,4 +86,4 @@ def test_simple_simulate_chunked(data, spec): choosers=data, spec=spec, nest_spec=None, chunk_size=2 ) expected = pd.Series([1, 1, 1], index=data.index) - pdt.assert_series_equal(choices, expected) + pdt.assert_series_equal(choices, expected, check_dtype=False) diff --git a/activitysim/core/test/test_timetable.py b/activitysim/core/test/test_timetable.py index 17b871dcfc..e8ac8e5555 100644 --- a/activitysim/core/test/test_timetable.py +++ b/activitysim/core/test/test_timetable.py @@ -7,6 +7,7 @@ import pandas as pd import pandas.testing as pdt import pytest +from numpy.testing import assert_array_equal from .. import chunk from .. import timetable as tt @@ -150,9 +151,8 @@ def test_basic(persons, tdd_alts): 17, # START + MIDDLE + END collides with same ] ) - pdt.assert_series_equal( - timetable.tour_available(person_ids, tdds), - pd.Series([True, True, False, False], index=person_ids.index), + assert_array_equal( + timetable.tour_available(person_ids, tdds), [True, True, False, False] ) # assigning overlapping trip END,START should convert END to START_END @@ -196,13 +196,17 @@ def test_basic(persons, tdd_alts): person_ids = pd.Series([0, 1, 2, 3, 4, 5]) periods = pd.Series([5, 5, 5, 5, 5, 5]) adjacent_run_length = timetable.adjacent_window_after(person_ids, periods) - pdt.assert_series_equal(adjacent_run_length, pd.Series([5, 5, 0, 5, 5, 3])) + pdt.assert_series_equal( + adjacent_run_length, pd.Series([5, 5, 0, 5, 5, 3]), check_dtype=False + ) # - adjacent_window_before person_ids = pd.Series([0, 1, 2, 3, 4, 5]) periods = pd.Series([10, 10, 10, 10, 10, 10]) adjacent_run_length = timetable.adjacent_window_before(person_ids, periods) - pdt.assert_series_equal(adjacent_run_length, pd.Series([5, 5, 1, 5, 5, 0])) + pdt.assert_series_equal( + adjacent_run_length, pd.Series([5, 5, 1, 5, 5, 0]), check_dtype=False + ) # - remaining_periods_available person_ids = pd.Series([0, 1, 2, 3]) diff --git a/activitysim/core/timetable.py b/activitysim/core/timetable.py index fe2aeee3cf..890be15d45 100644 --- a/activitysim/core/timetable.py +++ b/activitysim/core/timetable.py @@ -4,6 +4,7 @@ import logging from builtins import object, range +import numba as nb import numpy as np import pandas as pd @@ -37,7 +38,7 @@ ] COLLISION_LIST = [a + (b << I_BIT_SHIFT) for a, b in COLLISIONS] - +COLLISION_ARRAY = np.asarray(COLLISION_LIST) # str versions of time windows period states C_EMPTY = str(I_EMPTY) @@ -47,6 +48,161 @@ C_START_END = str(I_START_END) +@nb.njit +def _fast_tour_available( + tdds, + tdd_footprints, + window_row_ids, + window_row_ix__mapper, + self_windows, +): + """ + + Parameters + ---------- + tdds : array-like, shape (k) + tdd_footprints : array-like, shape (c, t) + window_row_ids : array-like, shape (k) + window_row_ix__mapper : FastMapping._mapper + self_windows : array-like + + Returns + ------- + array of bool, shape (k) + """ + out = np.ones_like(tdds, dtype=np.bool_) + for k in range(tdds.shape[0]): + tour_footprints = tdd_footprints[tdds[k]] # -> shape (t) + row_ix = window_row_ix__mapper[window_row_ids[k]] + windows = self_windows[row_ix] + x = tour_footprints + (windows << I_BIT_SHIFT) + stop = False + for j in range(COLLISION_ARRAY.size): + for i in range(x.size): + if x[i] == COLLISION_ARRAY[j]: + out[k] = False + stop = True + break + if stop: + break + return out + + +@nb.njit +def _available_run_length( + available, + before, + periods, + time_ix_mapper, +): + num_rows = available.shape[0] + num_cols = available.shape[1] + _time_col_ix_map = np.arange(num_cols) + available_run_length = np.zeros(num_rows, dtype=np.int32) + for row in range(num_rows): + _time_col_ix = time_ix_mapper[periods[row]] # scalar + if before: + mask = (_time_col_ix_map < _time_col_ix) * 1 + # index of first unavailable window after time + first_unavailable = np.where( + (1 - available[row]) * mask, _time_col_ix_map, 0 + ).max() + available_run_length[row] = _time_col_ix - first_unavailable - 1 + else: + # ones after specified time, zeroes before + mask = (_time_col_ix_map > _time_col_ix) * 1 + # index of first unavailable window after time + first_unavailable = np.where( + (1 - available[row]) * mask, _time_col_ix_map, num_cols + ).min() + available_run_length[row] = first_unavailable - _time_col_ix - 1 + return available_run_length + + +@nb.njit +def _available_run_length_1( + windows, + window_row_mapper, + time_ix_mapper, + before, + window_row_id, + period, +): + num_cols = windows.shape[1] + _time_col_ix_map = np.arange(num_cols) + available = np.ones(num_cols, dtype=np.int8) + available[0] = 0 + available[-1] = 0 + + window_row = windows[window_row_mapper[window_row_id], :] + for j in range(1, num_cols - 1): + if window_row[j] != I_MIDDLE: + available[j] = 1 + else: + available[j] = 0 + + _time_col_ix = time_ix_mapper[period] # scalar + if before: + mask = (_time_col_ix_map < _time_col_ix) * 1 + # index of first unavailable window after time + first_unavailable = np.where((1 - available) * mask, _time_col_ix_map, 0).max() + available_run_length = _time_col_ix - first_unavailable - 1 + else: + # ones after specified time, zeroes before + mask = (_time_col_ix_map > _time_col_ix) * 1 + # index of first unavailable window after time + first_unavailable = np.where( + (1 - available) * mask, _time_col_ix_map, num_cols + ).min() + available_run_length = first_unavailable - _time_col_ix - 1 + return available_run_length + + +@nb.njit +def _available_run_length_2( + windows, + window_row_mapper, + time_ix_mapper, + before, + window_row_id_values, + periods, +): + num_rows = window_row_id_values.shape[0] + num_cols = windows.shape[1] + _time_col_ix_map = np.arange(num_cols) + available_run_length = np.zeros(num_rows, dtype=np.int32) + available = np.ones(num_cols, dtype=np.int8) + available[0] = 0 + available[-1] = 0 + for row in range(num_rows): + + row_ix = window_row_mapper[window_row_id_values[row]] + window_row = windows[row_ix] + for j in range(1, num_cols - 1): + if window_row[j] != I_MIDDLE: + available[j] = 1 + else: + available[j] = 0 + + _time_col_ix = time_ix_mapper[periods[row]] # scalar + if before: + mask = (_time_col_ix_map < _time_col_ix) * 1 + # index of first unavailable window after time + first_unavailable = np.where( + (1 - available) * mask, _time_col_ix_map, 0 + ).max() + available_run_length[row] = _time_col_ix - first_unavailable - 1 + else: + # ones after specified time, zeroes before + mask = (_time_col_ix_map > _time_col_ix) * 1 + # index of first unavailable window after time + first_unavailable = np.where( + (1 - available) * mask, _time_col_ix_map, num_cols + ).min() + available_run_length[row] = first_unavailable - _time_col_ix - 1 + return available_run_length + + def tour_map(persons, tours, tdd_alts, persons_id_col="person_id"): sigil = { @@ -196,13 +352,15 @@ def __init__(self, windows_df, tdd_alts_df, table_name=None): self.checkpoint_df = None # series to map window row index value to window row's ordinal index - self.window_row_ix = pd.Series( - list(range(len(windows_df.index))), index=windows_df.index + from ..core.fast_mapping import FastMapping + + self.window_row_ix = FastMapping( + pd.Series(list(range(len(windows_df.index))), index=windows_df.index) ) int_time_periods = [int(c) for c in windows_df.columns.values] - self.time_ix = pd.Series( - list(range(len(windows_df.columns))), index=int_time_periods + self.time_ix = FastMapping( + pd.Series(list(range(len(windows_df.columns))), index=int_time_periods) ) # - pre-compute window state footprints for every tdd_alt @@ -246,12 +404,19 @@ def rollback(self): self.checkpoint_df = None self.transaction_loggers = None + def export_for_numba(self): + return dict( + tt_row_mapper=self.window_row_ix._mapper, + tt_col_mapper=self.time_ix._mapper, + tt_windows=self.windows, + ) + def slice_windows_by_row_id(self, window_row_ids): """ return windows array slice containing rows for specified window_row_ids (in window_row_ids order) """ - row_ixs = window_row_ids.map(self.window_row_ix).values + row_ixs = self.window_row_ix.apply_to(window_row_ids.values) windows = self.windows[row_ixs] return windows @@ -259,10 +424,10 @@ def slice_windows_by_row_id(self, window_row_ids): def slice_windows_by_row_id_and_period(self, window_row_ids, periods): # row ixs of tour_df group rows in windows - row_ixs = window_row_ids.map(self.window_row_ix).values + row_ixs = self.window_row_ix.apply_to(window_row_ids) # col ixs of periods in windows - time_col_ixs = periods.map(self.time_ix).values + time_col_ixs = self.time_ix.apply_to(periods) windows = self.windows[row_ixs, time_col_ixs] @@ -317,22 +482,40 @@ def tour_available(self, window_row_ids, tdds): available : pandas Series of bool with same index as window_row_ids.index (presumably tour_id, but we don't care) """ + if isinstance(tdds, pd.Series): + tdds = tdds.astype(int).to_numpy() + else: + tdds = tdds.astype(int) + if isinstance(window_row_ids, pd.Series): + window_row_ids = window_row_ids.astype(int).to_numpy() + else: + window_row_ids = window_row_ids.astype(int) + + available = _fast_tour_available( + tdds, + self.tdd_footprints, + window_row_ids, + self.window_row_ix._mapper, + self.windows, + ) - assert len(window_row_ids) == len(tdds) - - # numpy array with one tdd_footprints_df row for tdds - tour_footprints = self.tdd_footprints[tdds.values.astype(int)] - - # numpy array with one windows row for each person - windows = self.slice_windows_by_row_id(window_row_ids) - - # t0 = tracing.print_elapsed_time("slice_windows_by_row_id", t0, debug=True) - - x = tour_footprints + (windows << I_BIT_SHIFT) - - available = ~np.isin(x, COLLISION_LIST).any(axis=1) - available = pd.Series(available, index=window_row_ids.index) - + # assert len(window_row_ids) == len(tdds) + # + # # numpy array with one tdd_footprints_df row for tdds + # tour_footprints = self.tdd_footprints[tdds.values.astype(int)] + # + # # numpy array with one windows row for each person + # windows = self.slice_windows_by_row_id(window_row_ids) + # + # # t0 = tracing.print_elapsed_time("slice_windows_by_row_id", t0, debug=True) + # + # x = tour_footprints + (windows << I_BIT_SHIFT) + # + # available = ~np.isin(x, COLLISION_LIST).any(axis=1) + # if isinstance(window_row_ids, pd.Series): + # available = pd.Series(available, index=window_row_ids.index) + # elif isinstance(window_row_ids, xr.DataArray): + # available = xr.DataArray(available, dims=window_row_ids.dims, coords=window_row_ids.coords) return available def assign(self, window_row_ids, tdds): @@ -359,7 +542,7 @@ def assign(self, window_row_ids, tdds): tour_footprints = self.tdd_footprints[tdds.values.astype(int)] # row idxs of windows to assign to - row_ixs = window_row_ids.map(self.window_row_ix).values + row_ixs = self.window_row_ix.apply_to(window_row_ids) self.windows[row_ixs] = np.bitwise_or(self.windows[row_ixs], tour_footprints) @@ -396,7 +579,7 @@ def assign_subtour_mask(self, window_row_ids, tdds): tour_footprints = self.tdd_footprints[tdds.values.astype(int)] # row idxs of windows to assign to - row_ixs = window_row_ids.map(self.window_row_ix).values + row_ixs = self.window_row_ix.apply_to(window_row_ids) self.windows[row_ixs] = (tour_footprints == 0) * I_MIDDLE @@ -424,7 +607,7 @@ def assign_footprints(self, window_row_ids, footprints): assert len(window_row_ids.values) == len(np.unique(window_row_ids.values)) # row idxs of windows to assign to - row_ixs = window_row_ids.map(self.window_row_ix).values + row_ixs = self.window_row_ix.apply_to(window_row_ids) self.windows[row_ixs] = np.bitwise_or(self.windows[row_ixs], footprints) @@ -459,47 +642,31 @@ def adjacent_window_run_length(self, window_row_ids, periods, before): trace_label = "tt.adjacent_window_run_length" with chunk.chunk_log(trace_label): - - time_col_ixs = periods.map(self.time_ix).values - chunk.log_df(trace_label, "time_col_ixs", time_col_ixs) + available_run_length = _available_run_length_2( + self.windows, + self.window_row_ix._mapper, + self.time_ix._mapper, + before, + window_row_ids.values, + periods.to_numpy(), + ) # sliced windows with 1s where windows state is I_MIDDLE and 0s elsewhere - available = (self.slice_windows_by_row_id(window_row_ids) != I_MIDDLE) * 1 - chunk.log_df(trace_label, "available", available) - - # padding periods not available - available[:, 0] = 0 - available[:, -1] = 0 - - # column idxs of windows - num_rows, num_cols = available.shape - time_col_ix_map = np.tile(np.arange(0, num_cols), num_rows).reshape( - num_rows, num_cols - ) - # 0 1 2 3 4 5... - # 0 1 2 3 4 5... - # 0 1 2 3 4 5... - chunk.log_df(trace_label, "time_col_ix_map", time_col_ix_map) - - if before: - # ones after specified time, zeroes before - mask = (time_col_ix_map < time_col_ixs.reshape(num_rows, 1)) * 1 - # index of first unavailable window after time - first_unavailable = np.where( - (1 - available) * mask, time_col_ix_map, 0 - ).max(axis=1) - available_run_length = time_col_ixs - first_unavailable - 1 - else: - # ones after specified time, zeroes before - mask = (time_col_ix_map > time_col_ixs.reshape(num_rows, 1)) * 1 - # index of first unavailable window after time - first_unavailable = np.where( - (1 - available) * mask, time_col_ix_map, num_cols - ).min(axis=1) - available_run_length = first_unavailable - time_col_ixs - 1 - - chunk.log_df(trace_label, "mask", mask) - chunk.log_df(trace_label, "first_unavailable", first_unavailable) + # available = (self.slice_windows_by_row_id(window_row_ids) != I_MIDDLE) * 1 + # chunk.log_df(trace_label, "available", available) + # + # # padding periods not available + # available[:, 0] = 0 + # available[:, -1] = 0 + # + # available_run_length = _available_run_length( + # available, + # before, + # periods.to_numpy(), + # self.time_ix._mapper, + # ) + # + # max_available_run_length = available_run_length.max() chunk.log_df(trace_label, "available_run_length", available_run_length) return pd.Series(available_run_length, index=window_row_ids.index) @@ -645,21 +812,8 @@ def remaining_periods_available(self, window_row_ids, starts, ends): available : pandas Series int number periods available indexed by window_row_ids.index """ - - assert len(window_row_ids) == len(starts) - assert len(window_row_ids) == len(ends) - - available = (self.slice_windows_by_row_id(window_row_ids) != I_MIDDLE).sum( - axis=1 - ) - - # don't count time window padding at both ends of day - available -= 2 - - available -= np.clip((ends - starts - 1), a_min=0, a_max=None) - available = pd.Series(available, index=window_row_ids.index) - - return available + result = tt_remaining_periods_available(self, window_row_ids, starts, ends) + return result def max_time_block_available(self, window_row_ids): """ @@ -673,37 +827,393 @@ def max_time_block_available(self, window_row_ids): ------- pandas.Series with same index as window_row_ids, and integer max_run_length of """ + result = pd.Series( + _max_time_blocks_available_1( + self.window_row_ix._mapper, self.windows, np.asarray(window_row_ids) + ), + index=window_row_ids.index, + ) + return result + + +@nb.njit +def _max_time_block_available_1(windows_row): + """ + + Parameters + ---------- + tt_windows : array[int8], 1 dimension + Array of currently scheduled stuff + tt_row_mapper : numba.typed.Dict[int,int] + Maps value in `window_row_id` to row position in `tt_windows`. + window_row_id : int - # FIXME consider dedupe/redupe window_row_ids for performance - # as this may be called for alts with lots of duplicates (e.g. trip scheduling time pressure calculations) + Returns + ------- + + """ + max_block_avail = 0 + current_block = 0 + for i in range(1, windows_row.size - 1): + if windows_row[i] != I_MIDDLE: + current_block += 1 + else: + current_block = 0 + if current_block > max_block_avail: + max_block_avail = current_block + return max_block_avail + + +@nb.njit +def _max_time_blocks_available_1(tt_window_row_ix, tt_windows, window_row_ids): + max_blocks = np.zeros(window_row_ids.size, dtype=np.uint8) + # FIXME consider dedupe/redupe window_row_ids for performance + # as this may be called for alts with lots of duplicates (e.g. trip scheduling time pressure calculations) + for j in range(window_row_ids.size): + max_blocks[j] = _max_time_block_available_1( + tt_windows[tt_window_row_ix[window_row_ids[j]]] + ) + return max_blocks - # sliced windows with 1s where windows state is I_MIDDLE and 0s elsewhere - available = (self.slice_windows_by_row_id(window_row_ids) != I_MIDDLE) * 1 - # np.set_printoptions(edgeitems=25, linewidth = 180) - # print(f"self.slice_windows_by_row_id(window_row_ids)\n{self.slice_windows_by_row_id(window_row_ids)}") +@nb.njit +def sharrow_tt_max_time_block_available(tt_windows, tt_row_mapper, window_row_id): + return _max_time_block_available_1(tt_windows[tt_row_mapper[window_row_id]]) - # padding periods not available - available[:, 0] = 0 - available[:, -1] = 0 - diffs = np.diff( - available - ) # 1 at start of run of availables, -1 at end, 0 everywhere else - start_row_index, starts = np.asarray( - diffs > 0 - ).nonzero() # indices of run starts - end_row_index, ends = np.asarray(diffs < 0).nonzero() # indices of run ends - assert ( - start_row_index == end_row_index - ).all() # because bounded, expect same number of starts and ends +def tt_slice_windows_by_row_id(tt_window_row_ix, tt_windows, window_row_ids): + """ + return windows array slice containing rows for specified window_row_ids + (in window_row_ids order) + """ + row_ixs = tt_window_row_ix.apply_to(window_row_ids.values) + windows = tt_windows[row_ixs] + + return windows + + +@nb.njit +def _count_windows_that_are_not_middles( + window_row_id, # int + windows_mapper, # nb.Dict[int,int] + windows, # ndarray +): + row_ix = windows_mapper[window_row_id] + x = 0 + for i in range(windows.shape[1]): + if windows[row_ix, i] != I_MIDDLE: + x += 1 + return x + + +@nb.njit +def sharrow_tt_remaining_periods_available( + tt_windows, # ndarray + tt_row_mapper, # nb.Dict[int,int] + window_row_id, # int + starter, # int + ender, # int +): + """ + Number of periods remaining available after hypothetical scheduling - # run_lengths like availability but with run length at start of every run and zeros elsewhere - # (row_indices of starts and ends are aligned, so end - start is run_length) - run_lengths = np.zeros_like(available) - run_lengths[start_row_index, starts] = ends - starts + This is what's left after a new tour or trip from `starts` to + `ends` is hypothetically scheduled. - # we just want to know the the longest one for each window_row_id - max_run_lengths = run_lengths.max(axis=1) + Implements MTC TM1 @@remainingPeriodsAvailableAlt - return pd.Series(max_run_lengths, index=window_row_ids.index) + The start and end periods will always be available after + scheduling, so ignore them. The periods between start and end + must be currently unscheduled, so assume they will become + unavailable after scheduling this window. + + Parameters + ---------- + tt_windows : array[int8], 2 dimensions + Array of currently scheduled stuff + tt_row_mapper : numba.typed.Dict[int,int] + Maps value in the `window_row_ids` to row positions in `windows`. + window_row_id : int + An identifier for which window row to use. + starter : int + The starting period of the new tour that will block windows. + ender : int + The ending period of the new tour that will block windows. + + Returns + ------- + int + """ + available = _count_windows_that_are_not_middles( + window_row_id, + tt_row_mapper, + tt_windows, + ) + # don't count time window padding at both ends of day + available -= 2 + this_block = ender - starter - 1 + if this_block > 0: + available -= this_block + return available + + +@nb.njit +def _remaining_periods_available( + windows, # ndarray + windows_row_mapper, + window_row_ids, # ndarray[int] + starts, # ndarray[int] + ends, # ndarray[int] +): + """ + Number of periods remaining available after hypothetical scheduling + + This is what's left after a new tour or trip from `starts` to + `ends` is hypothetically scheduled. + + Implements MTC TM1 @@remainingPeriodsAvailableAlt + + The start and end periods will always be available after + scheduling, so ignore them. The periods between start and end + must be currently unscheduled, so assume they will become + unavailable after scheduling this window. + + Parameters + ---------- + windows_row_mapper : numba.typed.Dict[int,int] + Maps value in the `window_row_ids` to row positions in `windows`. + windows : array[int8], 2 dimensions + window_row_ids : array[int], 1-dimension + starts : array[int] + A 1-dimension array the same shape as `window_row_ids` which + gives the starting period of the new tour that will block windows. + ends : array[int], 1-dimension + A 1-dimension array the same shape as `window_row_ids` which + gives the ending period of the new tour that will block windows. + + Returns + ------- + array[int] + A 1-dimension array the same shape as `window_row_ids` which + gives the number of available periods remaining. + """ + result = np.empty(window_row_ids.shape, dtype=np.int64) + for i in range(window_row_ids.shape[0]): + result[i] = sharrow_tt_remaining_periods_available( + windows, + windows_row_mapper, + window_row_ids[i], + starts[i], + ends[i], + ) + return result + + +def tt_remaining_periods_available(tt, window_row_ids, starts, ends): + """ + Number of periods remaining available after hypothetical scheduling + + That is, what's left after something from starts to ends is + hypothetically scheduled + + Implements MTC TM1 @@remainingPeriodsAvailableAlt + + The start and end periods will always be available after + scheduling, so ignore them. The periods between start and end + must be currently unscheduled, so assume they will become + unavailable after scheduling this window. + + Parameters + ---------- + tt : TimeTable + window_row_ids : pandas.Series[int] + series of window_row_ids indexed by tour_id + starts : pandas.Series[int] + series of tdd_alt ids, index irrelevant (one per window_row_id) + ends : pandas.Series[int] + series of tdd_alt ids, index irrelevant (one per window_row_id) + + Returns + ------- + available : pandas Series int + number periods available indexed by window_row_ids.index + """ + + result = _remaining_periods_available( + tt.windows, + tt.window_row_ix._mapper, + window_row_ids.values, + starts.values, + ends.values, + ) + if isinstance(window_row_ids, pd.Series): + result = pd.Series(result, index=window_row_ids.index) + return result + + +@nb.njit +def _window_period_in_states( + windows, + windows_row_mapper, + windows_col_mapper, + window_row_id, + period, + state1, + state2, +): + """ + Return boolean indicating whether specified window periods are in list of states. + + Internal DRY method to implement previous_tour_ends and previous_tour_begins + + Parameters + ---------- + windows : array of int8, 2 dimensions + Array of currently scheduled stuff + windows_row_mapper : numba.typed.Dict[int,int] + Maps value in the `window_row_ids` to row positions in `windows`. + windows_col_mapper : numba.typed.Dict[int,int] + Array of currently scheduled stuff + window_row_id : int + An identifier for which window row to use. + period : int + An identifier for which window col to use. + state1, state2 : int + presumably (e.g. I_EMPTY, I_START...) + + Returns + ------- + bool + """ + w = windows[windows_row_mapper[window_row_id], windows_col_mapper[period]] + if w == state1 or w == state2: + return True + return False + + +@nb.njit +def _windows_periods_in_states( + windows, + windows_row_mapper, + windows_col_mapper, + window_row_ids, + periods, + state1, + state2, +): + result = np.empty(window_row_ids.shape, dtype=np.int8) + for i in range(window_row_ids.shape[0]): + result[i] = _window_period_in_states( + windows, + windows_row_mapper, + windows_col_mapper, + window_row_ids[i], + periods[i], + state1, + state2, + ) + return result + + +def tt_previous_tour_ends(tt, window_row_ids, periods): + return _windows_periods_in_states( + tt.windows, + tt.window_row_ix._mapper, + tt.time_ix._mapper, + window_row_ids.values, + periods.values, + I_END, + I_START_END, + ) + + +@nb.njit +def sharrow_tt_previous_tour_ends( + tt_windows, tt_row_mapper, tt_col_mapper, window_row_id, period +): + return _window_period_in_states( + tt_windows, + tt_row_mapper, + tt_col_mapper, + window_row_id, + period, + I_END, + I_START_END, + ) + + +def tt_previous_tour_begins(tt, window_row_ids, periods): + return _windows_periods_in_states( + tt.windows, + tt.window_row_ix._mapper, + tt.time_ix._mapper, + window_row_ids.values, + periods.values, + I_START, + I_START_END, + ) + + +@nb.njit +def sharrow_tt_previous_tour_begins( + tt_windows, tt_row_mapper, tt_col_mapper, window_row_id, period +): + return _window_period_in_states( + tt_windows, + tt_row_mapper, + tt_col_mapper, + window_row_id, + period, + I_START, + I_START_END, + ) + + +def tt_adjacent_window_before(tt, window_row_ids, periods): + return _available_run_length_2( + tt.windows, + tt.window_row_ix._mapper, + tt.time_ix._mapper, + True, + window_row_ids.values, + periods.to_numpy(), + ) + + +@nb.njit +def sharrow_tt_adjacent_window_before( + tt_windows, tt_row_mapper, tt_col_mapper, window_row_id, period +): + return _available_run_length_1( + tt_windows, + tt_row_mapper, + tt_col_mapper, + True, + window_row_id, + period, + ) + + +def tt_adjacent_window_after(tt, window_row_ids, periods): + return _available_run_length_2( + tt.windows, + tt.window_row_ix._mapper, + tt.time_ix._mapper, + False, + window_row_ids.values, + periods.to_numpy(), + ) + + +@nb.njit +def sharrow_tt_adjacent_window_after( + tt_windows, tt_row_mapper, tt_col_mapper, window_row_id, period +): + return _available_run_length_1( + tt_windows, + tt_row_mapper, + tt_col_mapper, + False, + window_row_id, + period, + ) diff --git a/activitysim/core/tracing.py b/activitysim/core/tracing.py index 22098065e1..bf7ae75a84 100644 --- a/activitysim/core/tracing.py +++ b/activitysim/core/tracing.py @@ -26,6 +26,8 @@ logger = logging.getLogger(__name__) +timing_notes = set() + class ElapsedTimeFormatter(logging.Formatter): def format(self, record): @@ -64,7 +66,8 @@ def print_elapsed_time(msg=None, t0=None, debug=False): return t1 -def log_runtime(model_name, start_time=None, timing=None): +def log_runtime(model_name, start_time=None, timing=None, force=False): + global timing_notes assert (start_time or timing) and not (start_time and timing) @@ -74,7 +77,7 @@ def log_runtime(model_name, start_time=None, timing=None): process_name = multiprocessing.current_process().name - if config.setting("multiprocess", False): + if config.setting("multiprocess", False) and not force: # when benchmarking, log timing for each processes in its own log if config.setting("benchmarking", False): header = "component_name,duration" @@ -86,9 +89,12 @@ def log_runtime(model_name, start_time=None, timing=None): if not inject.get_injectable("locutor", False): return - header = "process_name,model_name,seconds,minutes" + header = "process_name,model_name,seconds,minutes,notes" + note = " ".join(timing_notes) with config.open_log_file("timing_log.csv", "a", header) as log_file: - print(f"{process_name},{model_name},{seconds},{minutes}", file=log_file) + print(f"{process_name},{model_name},{seconds},{minutes},{note}", file=log_file) + + timing_notes.clear() def delete_output_files(file_type, ignore=None, subdir=None): @@ -833,7 +839,10 @@ def trace_interaction_eval_results(trace_results, trace_ids, label): slicer_column_name = trace_ids[0] - trace_results[slicer_column_name] = trace_ids[1] + try: + trace_results[slicer_column_name] = trace_ids[1] + except ValueError: + trace_results[slicer_column_name] = int(trace_ids[1]) targets = np.unique(trace_ids[1]) diff --git a/activitysim/core/util.py b/activitysim/core/util.py index 9a2f5c18d5..95ddb28d16 100644 --- a/activitysim/core/util.py +++ b/activitysim/core/util.py @@ -168,15 +168,12 @@ def reindex(series1, series2): """ - # turns out the merge is much faster than the .loc below - df = pd.merge( - series2.to_frame(name="left"), - series1.to_frame(name="right"), - left_on="left", - right_index=True, - how="left", - ) - return df.right + result = series1.reindex(series2) + try: + result.index = series2.index + except AttributeError: + pass + return result # return pd.Series(series1.loc[series2.values].values, index=series2.index) diff --git a/activitysim/estimation/test/test_larch_estimation/test_joint_tour_scheduling_loglike.csv b/activitysim/estimation/test/test_larch_estimation/test_joint_tour_scheduling_loglike.csv index cf0a9aeefa..a8e60cc389 100644 --- a/activitysim/estimation/test/test_larch_estimation/test_joint_tour_scheduling_loglike.csv +++ b/activitysim/estimation/test/test_larch_estimation/test_joint_tour_scheduling_loglike.csv @@ -1,2 +1,2 @@ ,loglike_prior,loglike_converge -0,-1295.265265023259,-173.82837615906058 +0,-1295.3015770898169,-175.64109886722656 diff --git a/activitysim/estimation/test/test_larch_estimation/test_mandatory_tour_scheduling_school_loglike.csv b/activitysim/estimation/test/test_larch_estimation/test_mandatory_tour_scheduling_school_loglike.csv index e113b20b42..e76a45b3fb 100644 --- a/activitysim/estimation/test/test_larch_estimation/test_mandatory_tour_scheduling_school_loglike.csv +++ b/activitysim/estimation/test/test_larch_estimation/test_mandatory_tour_scheduling_school_loglike.csv @@ -1,2 +1,2 @@ ,loglike_prior,loglike_converge -0,-2431.4832345854475,-2331.7641653716646 +0,-2475.6453326598094,-2344.7409183750583 diff --git a/activitysim/estimation/test/test_larch_estimation/test_mandatory_tour_scheduling_work_loglike.csv b/activitysim/estimation/test/test_larch_estimation/test_mandatory_tour_scheduling_work_loglike.csv index 7cf505c935..6970d3e666 100644 --- a/activitysim/estimation/test/test_larch_estimation/test_mandatory_tour_scheduling_work_loglike.csv +++ b/activitysim/estimation/test/test_larch_estimation/test_mandatory_tour_scheduling_work_loglike.csv @@ -1,2 +1,2 @@ ,loglike_prior,loglike_converge -0,-9139.1009551683674,-8399.8219678385103 +0,-9138.134506637969,-8404.2487574734187 diff --git a/activitysim/estimation/test/test_larch_estimation/test_non_mandatory_tour_scheduling_loglike.csv b/activitysim/estimation/test/test_larch_estimation/test_non_mandatory_tour_scheduling_loglike.csv index 3a5c071b3a..3282b45e71 100644 --- a/activitysim/estimation/test/test_larch_estimation/test_non_mandatory_tour_scheduling_loglike.csv +++ b/activitysim/estimation/test/test_larch_estimation/test_non_mandatory_tour_scheduling_loglike.csv @@ -1,2 +1,2 @@ ,loglike_prior,loglike_converge -0,-12768.066280736257,-9499.3997160849394 +0,-12772.911183835131,-9506.0541725800285 diff --git a/activitysim/estimation/test/test_larch_estimation/test_scheduling_model_joint_tour_scheduling_SLSQP_.csv b/activitysim/estimation/test/test_larch_estimation/test_scheduling_model_joint_tour_scheduling_SLSQP_.csv index b11e454f69..091e76401e 100644 --- a/activitysim/estimation/test/test_larch_estimation/test_scheduling_model_joint_tour_scheduling_SLSQP_.csv +++ b/activitysim/estimation/test/test_larch_estimation/test_scheduling_model_joint_tour_scheduling_SLSQP_.csv @@ -1,60 +1,60 @@ ,value,initvalue,nullvalue,minimum,maximum,best -coef_adjacent_window_exists_after_this_arrival_hour_first_tour_interaction,-13.52657987575712,-0.025700000000000001,0,-25,25,-13.52657987575712 -coef_adjacent_window_exists_after_this_arrival_hour_second_tour_interaction,-14.169340715131355,-0.02734,0,-25,25,-14.169340715131355 -coef_adjacent_window_exists_before_this_departure_hour_first_tour_interaction,-13.495922438017288,0.0084419999999999999,0,-25,25,-13.495922438017288 -coef_adjacent_window_exists_before_this_departure_hour_second_tour_interaction,-7.7162465780101046,-0.059299999999999999,0,-25,25,-7.7162465780101046 -coef_adult_with_children_in_hh_arrive_19_21,-0.0025406451956376816,0.33600000000000002,0,-25,25,-0.0025406451956376816 -coef_arrival_constants_am_peak,4.6476727087794183,-8.7288800000000002,0,-25,25,4.6476727087794183 -coef_arrival_constants_early,24.793142364658262,-8.7288800000000002,0,-25,25,24.793142364658262 -coef_arrival_constants_evening,-4.2917831342034436,-2.7489400000000002,0,-25,25,-4.2917831342034436 -coef_arrival_constants_late,-9.6170378774836092,-4.2425300000000004,0,-25,25,-9.6170378774836092 +coef_adjacent_window_exists_after_this_arrival_hour_first_tour_interaction,-0.028774249414067657,-0.025700000000000001,0,-25,25,-0.028774249414067657 +coef_adjacent_window_exists_after_this_arrival_hour_second_tour_interaction,0.91901966813332936,-0.02734,0,-25,25,0.91901966813332936 +coef_adjacent_window_exists_before_this_departure_hour_first_tour_interaction,0.03464166423412067,0.0084419999999999999,0,-25,25,0.03464166423412067 +coef_adjacent_window_exists_before_this_departure_hour_second_tour_interaction,-18.311237242705495,-0.059299999999999999,0,-25,25,-18.311237242705495 +coef_adult_with_children_in_hh_arrive_19_21,0.04436348324103874,0.33600000000000002,0,-25,25,0.04436348324103874 +coef_arrival_constants_am_peak,4.7614100686980576,-8.7288800000000002,0,-25,25,4.7614100686980576 +coef_arrival_constants_early,24.924968513710187,-8.7288800000000002,0,-25,25,24.924968513710187 +coef_arrival_constants_evening,-4.3200714359409025,-2.7489400000000002,0,-25,25,-4.3200714359409025 +coef_arrival_constants_late,-9.6327188982749732,-4.2425300000000004,0,-25,25,-9.6327188982749732 coef_arrival_constants_midday_1,0,0,0,0,0,0 -coef_arrival_constants_midday_2,2.6925192121364452,1.40804,0,-25,25,2.6925192121364452 -coef_arrival_constants_pm_peak_1,1.8492366917320286,1.0203599999999999,0,-25,25,1.8492366917320286 -coef_arrival_constants_pm_peak_2,1.5184104637159339,1.06863,0,-25,25,1.5184104637159339 +coef_arrival_constants_midday_2,2.6541662776744461,1.40804,0,-25,25,2.6541662776744461 +coef_arrival_constants_pm_peak_1,1.8388653479429438,1.0203599999999999,0,-25,25,1.8388653479429438 +coef_arrival_constants_pm_peak_2,1.4625561851708289,1.06863,0,-25,25,1.4625561851708289 coef_arrival_constants_pm_peak_3,0,0,0,0,0,0 -coef_arrival_constants_pm_peak_4,-1.1827680035278383,-0.59626000000000001,0,-25,25,-1.1827680035278383 +coef_arrival_constants_pm_peak_4,-1.2333276504506303,-0.59626000000000001,0,-25,25,-1.2333276504506303 coef_departure_constants_am_peak_1,-25,-11.595050000000001,0,-25,25,-25 -coef_departure_constants_am_peak_2,-4.2368491295195421,-9.0051900000000007,0,-25,25,-4.2368491295195421 -coef_departure_constants_am_peak_3,0.80622324441230619,-2.7331500000000002,0,-25,25,0.80622324441230619 -coef_departure_constants_am_peak_4,1.4081419130496329,0.26654,0,-25,25,1.4081419130496329 -coef_departure_constants_early,-24.418086041106228,-14.477080000000001,0,-25,25,-24.418086041106228 -coef_departure_constants_evening,-14.557517186260492,-18.987369999999999,0,-25,25,-14.557517186260492 -coef_departure_constants_late,-20.421814010799007,-20.27807,0,-25,25,-20.421814010799007 +coef_departure_constants_am_peak_2,-4.2487039295223656,-9.0051900000000007,0,-25,25,-4.2487039295223656 +coef_departure_constants_am_peak_3,0.75112055157913948,-2.7331500000000002,0,-25,25,0.75112055157913948 +coef_departure_constants_am_peak_4,1.3921313310397618,0.26654,0,-25,25,1.3921313310397618 +coef_departure_constants_early,-24.394585604972157,-14.477080000000001,0,-25,25,-24.394585604972157 +coef_departure_constants_evening,-14.12448484303447,-18.987369999999999,0,-25,25,-14.12448484303447 +coef_departure_constants_late,-19.873640270611908,-20.27807,0,-25,25,-19.873640270611908 coef_departure_constants_midday_1,0,0,0,0,0,0 -coef_departure_constants_midday_2,-4.5012376516121435,-1.6026,0,-25,25,-4.5012376516121435 -coef_departure_constants_pm_peak,-12.638402978350928,-17.695979999999999,0,-25,25,-12.638402978350928 -coef_destination_in_cbd_duration_shift_effects,0.57232969025524438,0.1067,0,-25,25,0.57232969025524438 -coef_discretionary_tour_duration_lt_2_hours,-1.0224876940373548,-0.69740000000000002,0,-25,25,-1.0224876940373548 -coef_duration_constants_0_to_1_hours,-2.7504813190145132,-2.2282600000000001,0,-25,25,-2.7504813190145132 -coef_duration_constants_11_to_13_hours,-6.1627365477909208,-0.81518999999999997,0,-25,25,-6.1627365477909208 -coef_duration_constants_14_to_18_hours,-2.8007056063538349,-2.7384400000000002,0,-25,25,-2.8007056063538349 +coef_departure_constants_midday_2,-4.3735430875047783,-1.6026,0,-25,25,-4.3735430875047783 +coef_departure_constants_pm_peak,-12.231456579430871,-17.695979999999999,0,-25,25,-12.231456579430871 +coef_destination_in_cbd_duration_shift_effects,0.56926682397665584,0.1067,0,-25,25,0.56926682397665584 +coef_discretionary_tour_duration_lt_2_hours,-0.96979691863694628,-0.69740000000000002,0,-25,25,-0.96979691863694628 +coef_duration_constants_0_to_1_hours,-2.7866437326016174,-2.2282600000000001,0,-25,25,-2.7866437326016174 +coef_duration_constants_11_to_13_hours,-5.6834374392880065,-0.81518999999999997,0,-25,25,-5.6834374392880065 +coef_duration_constants_14_to_18_hours,-2.7347130621577005,-2.7384400000000002,0,-25,25,-2.7347130621577005 coef_duration_constants_2_to_3_hours,0,0,0,0,0,0 -coef_duration_constants_4_to_5_hours,0.30114528727921769,-0.56174000000000002,0,-25,25,0.30114528727921769 -coef_duration_constants_6_to_7_hours,0.14407175378681686,-0.65547,0,-25,25,0.14407175378681686 -coef_duration_constants_8_to_10_hours,-10.570858039965891,-0.74061999999999995,0,-25,25,-10.570858039965891 -coef_eat_out_tour_departure_shift_effects,-0.024648889447639735,0.075490000000000002,0,-25,25,-0.024648889447639735 -coef_first_of_2_plus_tours_for_same_purpose_departure_shift_effect,-1.3293508348128713,-0.2364,0,-25,25,-1.3293508348128713 -coef_free_flow_round_trip_auto_time_shift_effects_duration,0.017752136350677013,0.0031949999999999999,0,-25,25,0.017752136350677013 -coef_maintenance_tour_depart_before_7,1.1545842619546469,-0.88260000000000005,0,-25,25,1.1545842619546469 -coef_maintenance_tour_departure_shift_effects,-0.15921016616264547,-0.1489,0,-25,25,-0.15921016616264547 -coef_maintenance_tour_duration_shift_effects,-0.14645897223903356,-0.083720000000000003,0,-25,25,-0.14645897223903356 -coef_number_of_joint_tours_departure_shift_effects,1.3561727548582434,0.052080000000000001,0,-25,25,1.3561727548582434 -coef_number_of_mandatory_tours_departure_shift_effects,0.076325869327416057,0.046730000000000001,0,-25,25,0.076325869327416057 +coef_duration_constants_4_to_5_hours,0.35927958946513583,-0.56174000000000002,0,-25,25,0.35927958946513583 +coef_duration_constants_6_to_7_hours,0.25829995673545386,-0.65547,0,-25,25,0.25829995673545386 +coef_duration_constants_8_to_10_hours,-10.443272918037978,-0.74061999999999995,0,-25,25,-10.443272918037978 +coef_eat_out_tour_departure_shift_effects,-0.026030105628492072,0.075490000000000002,0,-25,25,-0.026030105628492072 +coef_first_of_2_plus_tours_for_same_purpose_departure_shift_effect,-1.6735672025262123,-0.2364,0,-25,25,-1.6735672025262123 +coef_free_flow_round_trip_auto_time_shift_effects_duration,0.016957888892283637,0.0031949999999999999,0,-25,25,0.016957888892283637 +coef_maintenance_tour_depart_before_7,1.1216498838609328,-0.88260000000000005,0,-25,25,1.1216498838609328 +coef_maintenance_tour_departure_shift_effects,-0.16045036076756827,-0.1489,0,-25,25,-0.16045036076756827 +coef_maintenance_tour_duration_shift_effects,-0.15774061753063057,-0.083720000000000003,0,-25,25,-0.15774061753063057 +coef_number_of_joint_tours_departure_shift_effects,1.3426208902212988,0.052080000000000001,0,-25,25,1.3426208902212988 +coef_number_of_mandatory_tours_departure_shift_effects,0.073116531424716663,0.046730000000000001,0,-25,25,0.073116531424716663 coef_school_child_age_16_plus_departure_shift_effects,0.072660000000000002,0.072660000000000002,0,-25,25,0.072660000000000002 coef_school_child_age_16_plus_duration_shift_effects,0.20949999999999999,0.20949999999999999,0,-25,25,0.20949999999999999 -coef_school_child_age_under_16_departure_shift_effects,0.23900726126553401,0.04657,0,-25,25,0.23900726126553401 -coef_school_child_age_under_16_duration_shift_effects,-0.56786310052985978,0.32719999999999999,0,-25,25,-0.56786310052985978 -coef_school_child_under_16_arrive_after_22,-6.1472373285382957,-1.1799999999999999,0,-25,25,-6.1472373285382957 -coef_shopping_tour_arrive_after_22,-7.2210408448157928,-0.60270000000000001,0,-25,25,-7.2210408448157928 -coef_shopping_tour_depart_before_8,1.4584880566051988,-1.0369999999999999,0,-25,25,1.4584880566051988 -coef_shopping_tour_departure_shift_effects,-0.045633495185504684,-0.060150000000000002,0,-25,25,-0.045633495185504684 -coef_shopping_tour_duration_lt_2_hours,0.18942741647184094,0.51680000000000004,0,-25,25,0.18942741647184094 -coef_shopping_tour_duration_shift_effects,-0.35970475378100253,-0.1208,0,-25,25,-0.35970475378100253 -coef_some_previously_scheduled_tour_begins_in_this_arrival_hour,0.41197887481555961,-0.3992,0,-25,25,0.41197887481555961 -coef_some_previously_scheduled_tour_ends_in_this_departure_hour,0.28202909641105622,-0.45619999999999999,0,-25,25,0.28202909641105622 -coef_subsequent_of_2_plus_tours_for_same_purpose_duration_shift_effect,0.19126381823461158,-0.1731,0,-25,25,0.19126381823461158 +coef_school_child_age_under_16_departure_shift_effects,0.22437768613209164,0.04657,0,-25,25,0.22437768613209164 +coef_school_child_age_under_16_duration_shift_effects,-0.55690620785460743,0.32719999999999999,0,-25,25,-0.55690620785460743 +coef_school_child_under_16_arrive_after_22,-6.2179802555750401,-1.1799999999999999,0,-25,25,-6.2179802555750401 +coef_shopping_tour_arrive_after_22,-6.8841149173847738,-0.60270000000000001,0,-25,25,-6.8841149173847738 +coef_shopping_tour_depart_before_8,1.6329282274649815,-1.0369999999999999,0,-25,25,1.6329282274649815 +coef_shopping_tour_departure_shift_effects,-0.032901677188934476,-0.060150000000000002,0,-25,25,-0.032901677188934476 +coef_shopping_tour_duration_lt_2_hours,0.069480688387001235,0.51680000000000004,0,-25,25,0.069480688387001235 +coef_shopping_tour_duration_shift_effects,-0.41367749415560207,-0.1208,0,-25,25,-0.41367749415560207 +coef_some_previously_scheduled_tour_begins_in_this_arrival_hour,0.46823426868427825,-0.3992,0,-25,25,0.46823426868427825 +coef_some_previously_scheduled_tour_ends_in_this_departure_hour,0.26003373330967577,-0.45619999999999999,0,-25,25,0.26003373330967577 +coef_subsequent_of_2_plus_tours_for_same_purpose_duration_shift_effect,-0.015340339657372392,-0.1731,0,-25,25,-0.015340339657372392 coef_unavailable,-999,-999,0,-999,-999,-999 -coef_university_student_arrive_after_22,-9.1810949757532487,0.54659999999999997,0,-25,25,-9.1810949757532487 -coef_visit_tour_departure_shift_effects,-0.1095149196358212,0.096879999999999994,0,-25,25,-0.1095149196358212 -coef_visit_tour_duration_shift_effects,-0.12381401024065011,0.1638,0,-25,25,-0.12381401024065011 +coef_university_student_arrive_after_22,-9.1734571378829219,0.54659999999999997,0,-25,25,-9.1734571378829219 +coef_visit_tour_departure_shift_effects,-0.11236540231411092,0.096879999999999994,0,-25,25,-0.11236540231411092 +coef_visit_tour_duration_shift_effects,-0.13492097024550309,0.1638,0,-25,25,-0.13492097024550309 diff --git a/activitysim/estimation/test/test_larch_estimation/test_scheduling_model_mandatory_tour_scheduling_school_SLSQP_.csv b/activitysim/estimation/test/test_larch_estimation/test_scheduling_model_mandatory_tour_scheduling_school_SLSQP_.csv index b82e6aef3a..93c03c449c 100644 --- a/activitysim/estimation/test/test_larch_estimation/test_scheduling_model_mandatory_tour_scheduling_school_SLSQP_.csv +++ b/activitysim/estimation/test/test_larch_estimation/test_scheduling_model_mandatory_tour_scheduling_school_SLSQP_.csv @@ -1,56 +1,56 @@ ,value,initvalue,nullvalue,minimum,maximum,best -coef_adjacent_window_exists_after_this_arrival_hour_first_tour_interaction,-0.41422850345277012,-0.0030490000000000001,0,-25,25,-0.41422850345277012 -coef_adjacent_window_exists_after_this_arrival_hour_second_plus_tour_interaction,-0.6401855452227263,-0.52710000000000001,0,-25,25,-0.6401855452227263 -coef_adjacent_window_exists_before_this_departure_hour_first_tour_interaction,1.6569196041966843,0.089749999999999996,0,-25,25,1.6569196041966843 -coef_adjacent_window_exists_before_this_departure_hour_second_plus_tour_interaction,0.22572243506435116,-0.44,0,-25,25,0.22572243506435116 -coef_all_adults_ft_worker_duration,0.052335612401385102,0.10929999999999999,0,-25,25,0.052335612401385102 -coef_arrival_constants_am_peak,-2.2391622684315053,-2.4287183990000001,0,-25,25,-2.2391622684315053 -coef_arrival_constants_early,-1.0659800055630519,-2.4287183990000001,0,-25,25,-1.0659800055630519 -coef_arrival_constants_evening,-1.0333285070293219,-0.87014690400000005,0,-25,25,-1.0333285070293219 -coef_arrival_constants_late,-1.315909676881561,-1.7520004899999999,0,-25,25,-1.315909676881561 -coef_arrival_constants_midday_1,-1.3023302498849483,-1.237908768,0,-25,25,-1.3023302498849483 -coef_arrival_constants_midday_2,-0.55467258320436619,-0.53976893100000001,0,-25,25,-0.55467258320436619 +coef_adjacent_window_exists_after_this_arrival_hour_first_tour_interaction,11.716776044898809,-0.0030490000000000001,0,-25,25,11.716776044898809 +coef_adjacent_window_exists_after_this_arrival_hour_second_plus_tour_interaction,-0.58735581754454202,-0.52710000000000001,0,-25,25,-0.58735581754454202 +coef_adjacent_window_exists_before_this_departure_hour_first_tour_interaction,9.2169111822551333,0.089749999999999996,0,-25,25,9.2169111822551333 +coef_adjacent_window_exists_before_this_departure_hour_second_plus_tour_interaction,-15.186947397585952,-0.44,0,-25,25,-15.186947397585952 +coef_all_adults_ft_worker_duration,0.0083360428258746171,0.10929999999999999,0,-25,25,0.0083360428258746171 +coef_arrival_constants_am_peak,-2.2525597933199202,-2.4287183990000001,0,-25,25,-2.2525597933199202 +coef_arrival_constants_early,-1.0600927435131249,-2.4287183990000001,0,-25,25,-1.0600927435131249 +coef_arrival_constants_evening,-1.024968926440146,-0.87014690400000005,0,-25,25,-1.024968926440146 +coef_arrival_constants_late,-1.3622777646923283,-1.7520004899999999,0,-25,25,-1.3622777646923283 +coef_arrival_constants_midday_1,-1.3329442334964623,-1.237908768,0,-25,25,-1.3329442334964623 +coef_arrival_constants_midday_2,-0.56302790885535348,-0.53976893100000001,0,-25,25,-0.56302790885535348 coef_arrival_constants_pm_peak_1,0,0,0,0,0,0 -coef_arrival_constants_pm_peak_2,-0.44138742316895407,-0.38916924800000002,0,-25,25,-0.44138742316895407 -coef_arrival_constants_pm_peak_3,-0.28124697553233768,-0.198120349,0,-25,25,-0.28124697553233768 -coef_arrival_constants_pm_peak_4,-0.37036768408938353,-0.25362468399999999,0,-25,25,-0.37036768408938353 -coef_departure_constants_am_peak_1,-1.8826940382143509,-1.6176440560000001,0,-25,25,-1.8826940382143509 +coef_arrival_constants_pm_peak_2,-0.47120087260372584,-0.38916924800000002,0,-25,25,-0.47120087260372584 +coef_arrival_constants_pm_peak_3,-0.2889635829589125,-0.198120349,0,-25,25,-0.2889635829589125 +coef_arrival_constants_pm_peak_4,-0.42904089312882804,-0.25362468399999999,0,-25,25,-0.42904089312882804 +coef_departure_constants_am_peak_1,-1.8402276885639066,-1.6176440560000001,0,-25,25,-1.8402276885639066 coef_departure_constants_am_peak_2,0,0,0,0,0,0 -coef_departure_constants_am_peak_3,-0.18398491652900856,-0.073826841000000004,0,-25,25,-0.18398491652900856 -coef_departure_constants_am_peak_4,-2.0164025628346165,-2.0805707689999999,0,-25,25,-2.0164025628346165 -coef_departure_constants_early,-3.6367697723264993,-3.8206624040000001,0,-25,25,-3.6367697723264993 -coef_departure_constants_evening,-5.2587949264840343,-5.2302878359999996,0,-25,25,-5.2587949264840343 -coef_departure_constants_late,-21.447622138036181,-11.88604728,0,-25,25,-21.447622138036181 -coef_departure_constants_midday_1,-3.2798639404148568,-2.9857394570000002,0,-25,25,-3.2798639404148568 -coef_departure_constants_midday_2,-3.7505480975849346,-3.6284346460000001,0,-25,25,-3.7505480975849346 -coef_departure_constants_pm_peak,-3.0931673015219667,-3.1025051499999998,0,-25,25,-3.0931673015219667 -coef_duration_constants_0_to_2_hours,-1.3721102503108922,-1.409955689,0,-25,25,-1.3721102503108922 -coef_duration_constants_10_hours,-0.87644803457973108,-0.90478898299999999,0,-25,25,-0.87644803457973108 -coef_duration_constants_11_hours,-1.1272644347425629,-1.5211626039999999,0,-25,25,-1.1272644347425629 -coef_duration_constants_12_to_13_hours,-2.6683455157140479,-2.4184889169999999,0,-25,25,-2.6683455157140479 -coef_duration_constants_14_to_18_hours,-2.6759742908121606,-2.5031372950000002,0,-25,25,-2.6759742908121606 -coef_duration_constants_3_to_4_hours,-0.55220033048204353,-0.74589325200000001,0,-25,25,-0.55220033048204353 -coef_duration_constants_5_to_6_hours,-0.53262716577049274,-0.56763662199999998,0,-25,25,-0.53262716577049274 +coef_departure_constants_am_peak_3,-0.14762834941122654,-0.073826841000000004,0,-25,25,-0.14762834941122654 +coef_departure_constants_am_peak_4,-2.0052907227909373,-2.0805707689999999,0,-25,25,-2.0052907227909373 +coef_departure_constants_early,-3.6385102149182318,-3.8206624040000001,0,-25,25,-3.6385102149182318 +coef_departure_constants_evening,-5.1709311530899091,-5.2302878359999996,0,-25,25,-5.1709311530899091 +coef_departure_constants_late,-25,-11.88604728,0,-25,25,-25 +coef_departure_constants_midday_1,-3.235655391222839,-2.9857394570000002,0,-25,25,-3.235655391222839 +coef_departure_constants_midday_2,-3.7122959166670451,-3.6284346460000001,0,-25,25,-3.7122959166670451 +coef_departure_constants_pm_peak,-2.9909526474284074,-3.1025051499999998,0,-25,25,-2.9909526474284074 +coef_duration_constants_0_to_2_hours,-1.3817519840859342,-1.409955689,0,-25,25,-1.3817519840859342 +coef_duration_constants_10_hours,-0.85439568067710703,-0.90478898299999999,0,-25,25,-0.85439568067710703 +coef_duration_constants_11_hours,-1.0970644700531507,-1.5211626039999999,0,-25,25,-1.0970644700531507 +coef_duration_constants_12_to_13_hours,-2.6774099148644868,-2.4184889169999999,0,-25,25,-2.6774099148644868 +coef_duration_constants_14_to_18_hours,-2.6488356018470931,-2.5031372950000002,0,-25,25,-2.6488356018470931 +coef_duration_constants_3_to_4_hours,-0.56119226026187308,-0.74589325200000001,0,-25,25,-0.56119226026187308 +coef_duration_constants_5_to_6_hours,-0.52869720683509602,-0.56763662199999998,0,-25,25,-0.52869720683509602 coef_duration_constants_7_to_8_hours,0,0,0,0,0,0 -coef_duration_constants_9_hours,-0.48760897429357175,-0.65080668399999997,0,-25,25,-0.48760897429357175 -coef_first_of_2plus_school_lt_6_hours,0.16905521319272807,1.4870000000000001,0,-25,25,0.16905521319272807 -coef_first_of_2plus_school_tours_departure,-0.3338303791406505,-0.30020000000000002,0,-25,25,-0.3338303791406505 -coef_first_of_2plus_school_tours_duration,-0.042558365362996578,-0.1593,0,-25,25,-0.042558365362996578 +coef_duration_constants_9_hours,-0.47578654406256893,-0.65080668399999997,0,-25,25,-0.47578654406256893 +coef_first_of_2plus_school_lt_6_hours,0.36465272850981928,1.4870000000000001,0,-25,25,0.36465272850981928 +coef_first_of_2plus_school_tours_departure,-0.28345842741237504,-0.30020000000000002,0,-25,25,-0.28345842741237504 +coef_first_of_2plus_school_tours_duration,-0.0072205064801072029,-0.1593,0,-25,25,-0.0072205064801072029 coef_ft_worker_departure,0.39710000000000001,0.39710000000000001,0,-25,25,0.39710000000000001 coef_ft_worker_duration,-0.1908,-0.1908,0,-25,25,-0.1908 -coef_hh_income_early_departure,-13.757852371659974,-0.88370000000000004,0,-25,25,-13.757852371659974 -coef_hh_income_late_arrival,0.10172645562322212,-0.3533,0,-25,25,0.10172645562322212 -coef_mode_choice_logsum,0.056409818603392878,2.1269999999999998,0,-25,25,0.056409818603392878 +coef_hh_income_early_departure,-12.781415662231149,-0.88370000000000004,0,-25,25,-12.781415662231149 +coef_hh_income_late_arrival,0.13481375603358139,-0.3533,0,-25,25,0.13481375603358139 +coef_mode_choice_logsum,-0.15485890248282916,2.1269999999999998,0,-25,25,-0.15485890248282916 coef_non_worker_departure,0.55389999999999995,0.55389999999999995,0,-25,25,0.55389999999999995 -coef_previous_tour_begins_this_arrival_hour,-10.370632174066369,-1.1020000000000001,0,-25,25,-10.370632174066369 -coef_previous_tour_ends_this_departure_hour,0.29855098378969142,-0.59950000000000003,0,-25,25,0.29855098378969142 -coef_remaining_work_school_tours_to_be_scheduled_div_number_of_unscheduled_hours,-25,-16.670000000000002,0,-25,25,-25 -coef_roundtrip_auto_time_to_work,0.005137149637746443,0.0031949999999999999,0,-25,25,0.005137149637746443 -coef_school_plus_work_tours_by_student_lt_6_hours,-2.4833134339994589,1.73,0,-25,25,-2.4833134339994589 -coef_school_plus_work_tours_by_worker_lt_6_hours,-2.0713134339992108,2.1419999999999999,0,-25,25,-2.0713134339992108 -coef_student_driver_duration,-0.0039575686674964382,0.034639999999999997,0,-25,25,-0.0039575686674964382 -coef_subsequent_2plus_school_tours_duration,-0.18829905225489668,-0.23380000000000001,0,-25,25,-0.18829905225489668 -coef_subsequent_of_2plus_school_lt_6_hours,25,2.1419999999999999,0,-25,25,25 +coef_previous_tour_begins_this_arrival_hour,-5.5053579900953125,-1.1020000000000001,0,-25,25,-5.5053579900953125 +coef_previous_tour_ends_this_departure_hour,-15.038466619861389,-0.59950000000000003,0,-25,25,-15.038466619861389 +coef_remaining_work_school_tours_to_be_scheduled_div_number_of_unscheduled_hours,-24.999999999999979,-16.670000000000002,0,-25,25,-24.999999999999979 +coef_roundtrip_auto_time_to_work,0.0053019952333040361,0.0031949999999999999,0,-25,25,0.0053019952333040361 +coef_school_plus_work_tours_by_student_lt_6_hours,-1.1660913354672906,1.73,0,-25,25,-1.1660913354672906 +coef_school_plus_work_tours_by_worker_lt_6_hours,-0.75409133546729057,2.1419999999999999,0,-25,25,-0.75409133546729057 +coef_student_driver_duration,-0.0068046815074113429,0.034639999999999997,0,-25,25,-0.0068046815074113429 +coef_subsequent_2plus_school_tours_duration,-0.21896654709477803,-0.23380000000000001,0,-25,25,-0.21896654709477803 +coef_subsequent_of_2plus_school_lt_6_hours,24.999999999999982,2.1419999999999999,0,-25,25,24.999999999999982 coef_subsequent_tour_must_start_after_previous_tour_ends,-100,-100,0,-100,-100,-100 -coef_univ_departure,0.23436737071396405,0.28000000000000003,0,-25,25,0.23436737071396405 -coef_univ_duration,-0.4091323980277653,-0.29070000000000001,0,-25,25,-0.4091323980277653 +coef_univ_departure,0.22263900806749343,0.28000000000000003,0,-25,25,0.22263900806749343 +coef_univ_duration,-0.41672805296299825,-0.29070000000000001,0,-25,25,-0.41672805296299825 diff --git a/activitysim/estimation/test/test_larch_estimation/test_scheduling_model_mandatory_tour_scheduling_work_SLSQP_.csv b/activitysim/estimation/test/test_larch_estimation/test_scheduling_model_mandatory_tour_scheduling_work_SLSQP_.csv index e101827da9..9465192ef9 100644 --- a/activitysim/estimation/test/test_larch_estimation/test_scheduling_model_mandatory_tour_scheduling_work_SLSQP_.csv +++ b/activitysim/estimation/test/test_larch_estimation/test_scheduling_model_mandatory_tour_scheduling_work_SLSQP_.csv @@ -1,64 +1,64 @@ ,value,initvalue,nullvalue,minimum,maximum,best -coef_adjacent_window_exists_after_this_arrival_hour_first_tour_interaction,0.32667520282267543,0.36270000000000002,0,-25,25,0.32667520282267543 -coef_adjacent_window_exists_after_this_arrival_hour_second_plus_tour_interaction,0.2989032139409768,-0.1012,0,-25,25,0.2989032139409768 -coef_adjacent_window_exists_before_this_departure_hour_first_tour_interaction,-0.038388513238313653,0.17710000000000001,0,-25,25,-0.038388513238313653 -coef_adjacent_window_exists_before_this_departure_hour_second_plus_tour_interaction,-0.012603680222033091,-0.21229999999999999,0,-25,25,-0.012603680222033091 -coef_arrival_constants_am_peak,-2.5271385106886068,-1.854520626,0,-25,25,-2.5271385106886068 +coef_adjacent_window_exists_after_this_arrival_hour_first_tour_interaction,25,0.36270000000000002,0,-25,25,25 +coef_adjacent_window_exists_after_this_arrival_hour_second_plus_tour_interaction,0.67172411345437255,-0.1012,0,-25,25,0.67172411345437255 +coef_adjacent_window_exists_before_this_departure_hour_first_tour_interaction,-0.2705642011106818,0.17710000000000001,0,-25,25,-0.2705642011106818 +coef_adjacent_window_exists_before_this_departure_hour_second_plus_tour_interaction,2.3126663890343506,-0.21229999999999999,0,-25,25,2.3126663890343506 +coef_arrival_constants_am_peak,-2.5608583720316691,-1.854520626,0,-25,25,-2.5608583720316691 coef_arrival_constants_early,0,0,0,0,0,0 -coef_arrival_constants_evening,-0.085796024190049405,0.103566251,0,-25,25,-0.085796024190049405 -coef_arrival_constants_late,-1.2214089672341761,-0.96595733900000003,0,-25,25,-1.2214089672341761 -coef_arrival_constants_midday_1,-0.48973614520419595,-0.49597203699999998,0,-25,25,-0.48973614520419595 -coef_arrival_constants_midday_2,-0.38325967338204053,-0.37855408099999999,0,-25,25,-0.38325967338204053 +coef_arrival_constants_evening,-0.11313543070951976,0.103566251,0,-25,25,-0.11313543070951976 +coef_arrival_constants_late,-1.2047708015331426,-0.96595733900000003,0,-25,25,-1.2047708015331426 +coef_arrival_constants_midday_1,-0.4666152062959858,-0.49597203699999998,0,-25,25,-0.4666152062959858 +coef_arrival_constants_midday_2,-0.37297675905912303,-0.37855408099999999,0,-25,25,-0.37297675905912303 coef_arrival_constants_pm_peak_1,0,0,0,0,0,0 -coef_arrival_constants_pm_peak_2,0.20200494978830774,0.27608389999999999,0,-25,25,0.20200494978830774 -coef_arrival_constants_pm_peak_3,0.57252378326592945,0.699587132,0,-25,25,0.57252378326592945 -coef_arrival_constants_pm_peak_4,0.70559147854432169,0.79928937700000002,0,-25,25,0.70559147854432169 -coef_departure_constants_am_peak_1,-0.6418973650064711,-0.61618090599999997,0,-25,25,-0.6418973650064711 +coef_arrival_constants_pm_peak_2,0.20356469833981647,0.27608389999999999,0,-25,25,0.20356469833981647 +coef_arrival_constants_pm_peak_3,0.55940049285496218,0.699587132,0,-25,25,0.55940049285496218 +coef_arrival_constants_pm_peak_4,0.69955932869966053,0.79928937700000002,0,-25,25,0.69955932869966053 +coef_departure_constants_am_peak_1,-0.65012576984935755,-0.61618090599999997,0,-25,25,-0.65012576984935755 coef_departure_constants_am_peak_2,0,0,0,0,0,0 -coef_departure_constants_am_peak_3,-0.32300636563617879,-0.254714726,0,-25,25,-0.32300636563617879 -coef_departure_constants_am_peak_4,-1.220041820613047,-1.2513460240000001,0,-25,25,-1.220041820613047 -coef_departure_constants_early,-0.88937510260312602,-0.95272526999999996,0,-25,25,-0.88937510260312602 -coef_departure_constants_evening,-1.9893998781318789,-1.610513243,0,-25,25,-1.9893998781318789 -coef_departure_constants_late,-2.9366745684271129,-2.8834152230000001,0,-25,25,-2.9366745684271129 -coef_departure_constants_midday_1,-1.8267001521037016,-1.7058689920000001,0,-25,25,-1.8267001521037016 -coef_departure_constants_midday_2,-1.8787827704964197,-1.6935705830000001,0,-25,25,-1.8787827704964197 -coef_departure_constants_pm_peak,-1.6918462289474312,-1.4399919619999999,0,-25,25,-1.6918462289474312 -coef_destination_in_cbd_departure_shift_effects,0.028678306805682247,0.047169999999999997,0,-25,25,0.028678306805682247 -coef_destination_in_cbd_duration_shift_effects,0.096673084899908696,0.086790000000000006,0,-25,25,0.096673084899908696 -coef_destination_in_cbd_early_departure_interaction,-0.41995880475080311,-0.45660000000000001,0,-25,25,-0.41995880475080311 -coef_destination_in_cbd_late_arrival_interaction,-0.26693459750183979,-0.2334,0,-25,25,-0.26693459750183979 -coef_duration_constants_0_to_2_hours,-2.0624023323687144,-2.5282663900000002,0,-25,25,-2.0624023323687144 +coef_departure_constants_am_peak_3,-0.32740035887947389,-0.254714726,0,-25,25,-0.32740035887947389 +coef_departure_constants_am_peak_4,-1.2190845361495699,-1.2513460240000001,0,-25,25,-1.2190845361495699 +coef_departure_constants_early,-0.92470832388900248,-0.95272526999999996,0,-25,25,-0.92470832388900248 +coef_departure_constants_evening,-1.8906229365299492,-1.610513243,0,-25,25,-1.8906229365299492 +coef_departure_constants_late,-2.9986176789858154,-2.8834152230000001,0,-25,25,-2.9986176789858154 +coef_departure_constants_midday_1,-1.8241536006077277,-1.7058689920000001,0,-25,25,-1.8241536006077277 +coef_departure_constants_midday_2,-1.8783523324805591,-1.6935705830000001,0,-25,25,-1.8783523324805591 +coef_departure_constants_pm_peak,-1.6396663560770837,-1.4399919619999999,0,-25,25,-1.6396663560770837 +coef_destination_in_cbd_departure_shift_effects,0.04109611455586798,0.047169999999999997,0,-25,25,0.04109611455586798 +coef_destination_in_cbd_duration_shift_effects,0.10130692865710192,0.086790000000000006,0,-25,25,0.10130692865710192 +coef_destination_in_cbd_early_departure_interaction,-0.38984353468069122,-0.45660000000000001,0,-25,25,-0.38984353468069122 +coef_destination_in_cbd_late_arrival_interaction,-0.27055298689359897,-0.2334,0,-25,25,-0.27055298689359897 +coef_duration_constants_0_to_2_hours,-2.1189652588431978,-2.5282663900000002,0,-25,25,-2.1189652588431978 coef_duration_constants_10_hours,0,0,0,0,0,0 -coef_duration_constants_11_hours,-0.35847896432255638,-0.34779539100000001,0,-25,25,-0.35847896432255638 -coef_duration_constants_12_to_13_hours,-1.1320004478276953,-1.0082223459999999,0,-25,25,-1.1320004478276953 -coef_duration_constants_14_to_18_hours,-1.8548759254550073,-1.701858847,0,-25,25,-1.8548759254550073 -coef_duration_constants_3_to_4_hours,-0.60027688746955321,-0.91897445700000002,0,-25,25,-0.60027688746955321 -coef_duration_constants_5_to_6_hours,-0.3435610046131029,-0.71855028799999998,0,-25,25,-0.3435610046131029 -coef_duration_constants_7_to_8_hours,-0.033979868351848914,-0.139623566,0,-25,25,-0.033979868351848914 -coef_duration_constants_9_hours,0.13725128186580737,0.055706243000000003,0,-25,25,0.13725128186580737 -coef_first_of_2plus_work_tours_departure_shift_effects,-0.2896837904560684,-0.30330000000000001,0,-25,25,-0.2896837904560684 -coef_first_of_2plus_work_tours_duration_lt_8_hrs,1.9167861172628315,1.98,0,-25,25,1.9167861172628315 -coef_first_of_2plus_work_tours_duration_shift_effects,-0.28618628658489031,-0.18609999999999999,0,-25,25,-0.28618628658489031 -coef_free_flow_round_trip_auto_time_shift_effects_departure,0.00059861431451224852,-0.00114,0,-25,25,0.00059861431451224852 -coef_free_flow_round_trip_auto_time_shift_effects_duration,0.0042157097953458792,0.0022100000000000002,0,-25,25,0.0042157097953458792 -coef_full_time_worker_10_to_12_departure_interaction,-0.67825091233923496,-0.51819999999999999,0,-25,25,-0.67825091233923496 -coef_full_time_worker_duration_lt_9_hours_interaction,-1.3572941093466813,-1.2569999999999999,0,-25,25,-1.3572941093466813 -coef_household_income_departure_shift_effects,0.00011131783637255824,0.00020799999999999999,0,-25,25,0.00011131783637255824 -coef_household_income_early_departure_interaction,-0.58277482301281736,-0.4854,0,-25,25,-0.58277482301281736 -coef_household_income_late_arrival_interaction,0.02995300537436358,-0.38390000000000002,0,-25,25,0.02995300537436358 -coef_mode_choice_logsum,0.24711457366198325,1.0269999999999999,0,-25,25,0.24711457366198325 +coef_duration_constants_11_hours,-0.35443610468830039,-0.34779539100000001,0,-25,25,-0.35443610468830039 +coef_duration_constants_12_to_13_hours,-1.1153352032051054,-1.0082223459999999,0,-25,25,-1.1153352032051054 +coef_duration_constants_14_to_18_hours,-1.8418267064323726,-1.701858847,0,-25,25,-1.8418267064323726 +coef_duration_constants_3_to_4_hours,-0.63573505547017051,-0.91897445700000002,0,-25,25,-0.63573505547017051 +coef_duration_constants_5_to_6_hours,-0.35577473775309831,-0.71855028799999998,0,-25,25,-0.35577473775309831 +coef_duration_constants_7_to_8_hours,-0.033757586303302736,-0.139623566,0,-25,25,-0.033757586303302736 +coef_duration_constants_9_hours,0.12934760728519379,0.055706243000000003,0,-25,25,0.12934760728519379 +coef_first_of_2plus_work_tours_departure_shift_effects,-0.27106753260112321,-0.30330000000000001,0,-25,25,-0.27106753260112321 +coef_first_of_2plus_work_tours_duration_lt_8_hrs,1.8948098471889567,1.98,0,-25,25,1.8948098471889567 +coef_first_of_2plus_work_tours_duration_shift_effects,-0.29362220747793244,-0.18609999999999999,0,-25,25,-0.29362220747793244 +coef_free_flow_round_trip_auto_time_shift_effects_departure,0.00054374628944530747,-0.00114,0,-25,25,0.00054374628944530747 +coef_free_flow_round_trip_auto_time_shift_effects_duration,0.0040398779948456849,0.0022100000000000002,0,-25,25,0.0040398779948456849 +coef_full_time_worker_10_to_12_departure_interaction,-0.66614758973349719,-0.51819999999999999,0,-25,25,-0.66614758973349719 +coef_full_time_worker_duration_lt_9_hours_interaction,-1.3694302807642016,-1.2569999999999999,0,-25,25,-1.3694302807642016 +coef_household_income_departure_shift_effects,0.00011200417166098945,0.00020799999999999999,0,-25,25,0.00011200417166098945 +coef_household_income_early_departure_interaction,-0.57489507784478577,-0.4854,0,-25,25,-0.57489507784478577 +coef_household_income_late_arrival_interaction,0.036426020508396037,-0.38390000000000002,0,-25,25,0.036426020508396037 +coef_mode_choice_logsum,0.20613495321472861,1.0269999999999999,0,-25,25,0.20613495321472861 coef_non_working_adult_duration_shift_effects,-0.1207,-0.1207,0,-25,25,-0.1207 -coef_part_time_worker_13_to_15_arrival_interaction,0.37532108740462561,0.54330000000000001,0,-25,25,0.37532108740462561 -coef_part_time_worker_departure_shift_effects,0.09509355721343847,0.067360000000000003,0,-25,25,0.09509355721343847 -coef_previously_scheduled_tour_begins_in_this_arrival_hour,-6.7245570005227204,-1.3340000000000001,0,-25,25,-6.7245570005227204 -coef_previously_scheduled_tour_ends_in_this_departure_hour,0.2588531712397476,-0.89349999999999996,0,-25,25,0.2588531712397476 -coef_remaining_tours_to_be_scheduled_div_number_of_unscheduled_hours,-6.8650722180171249,-18.68,0,-25,25,-6.8650722180171249 +coef_part_time_worker_13_to_15_arrival_interaction,0.33677938786979689,0.54330000000000001,0,-25,25,0.33677938786979689 +coef_part_time_worker_departure_shift_effects,0.089714285580281661,0.067360000000000003,0,-25,25,0.089714285580281661 +coef_previously_scheduled_tour_begins_in_this_arrival_hour,-6.6033938783133914,-1.3340000000000001,0,-25,25,-6.6033938783133914 +coef_previously_scheduled_tour_ends_in_this_departure_hour,2.4216789099176217,-0.89349999999999996,0,-25,25,2.4216789099176217 +coef_remaining_tours_to_be_scheduled_div_number_of_unscheduled_hours,-6.3408961734536069,-18.68,0,-25,25,-6.3408961734536069 coef_rural_household_early_departure_interaction,0.40389999999999998,0.40389999999999998,0,-25,25,0.40389999999999998 coef_rural_household_late_arrival_interaction,-0.34510000000000002,-0.34510000000000002,0,-25,25,-0.34510000000000002 -coef_subsequent_2plus_work_departure_tours_shift_effects,0.060824095014172706,-0.53810000000000002,0,-25,25,0.060824095014172706 -coef_subsequent_2plus_work_duration_tours_shift_effects,-0.08095682334415219,-0.31740000000000002,0,-25,25,-0.08095682334415219 -coef_subsequent_of_2plus_work_tours_duration_lt_8_hrs,5.4433872561126897,2.5819999999999999,0,-25,25,5.4433872561126897 +coef_subsequent_2plus_work_departure_tours_shift_effects,0.015346535529312815,-0.53810000000000002,0,-25,25,0.015346535529312815 +coef_subsequent_2plus_work_duration_tours_shift_effects,-0.12773109887960765,-0.31740000000000002,0,-25,25,-0.12773109887960765 +coef_subsequent_of_2plus_work_tours_duration_lt_8_hrs,3.5917959509926001,2.5819999999999999,0,-25,25,3.5917959509926001 coef_subsequent_tour_must_start_after_previous_tour_ends,-100,-100,0,-100,-100,-100 -coef_tours_by_student_duration_lt_8_hrs,20.916124133444324,2.5819999999999999,0,-25,25,20.916124133444324 -coef_tours_by_worker_duration_lt_8_hrs,19.246724133444353,0.91259999999999997,0,-25,25,19.246724133444353 -coef_university_student_departure_shift_effects,0.076480195754151201,0.05747,0,-25,25,0.076480195754151201 +coef_tours_by_student_duration_lt_8_hrs,14.705420185896203,2.5819999999999999,0,-25,25,14.705420185896203 +coef_tours_by_worker_duration_lt_8_hrs,13.036020185896209,0.91259999999999997,0,-25,25,13.036020185896209 +coef_university_student_departure_shift_effects,0.052139288632078368,0.05747,0,-25,25,0.052139288632078368 diff --git a/activitysim/estimation/test/test_larch_estimation/test_scheduling_model_non_mandatory_tour_scheduling_SLSQP_.csv b/activitysim/estimation/test/test_larch_estimation/test_scheduling_model_non_mandatory_tour_scheduling_SLSQP_.csv index e733c6608d..68cc071e89 100644 --- a/activitysim/estimation/test/test_larch_estimation/test_scheduling_model_non_mandatory_tour_scheduling_SLSQP_.csv +++ b/activitysim/estimation/test/test_larch_estimation/test_scheduling_model_non_mandatory_tour_scheduling_SLSQP_.csv @@ -1,90 +1,90 @@ ,value,initvalue,nullvalue,minimum,maximum,best -coef_adjacent_window_exists_after_this_arrival_hour_first_tour_interaction,-0.12795390836606826,-0.025700000000000001,0,-25,25,-0.12795390836606826 -coef_adjacent_window_exists_after_this_arrival_hour_second_plus_tour_interaction,0.093953618800295624,-0.02734,0,-25,25,0.093953618800295624 -coef_adjacent_window_exists_before_this_departure_hour_first_tour_interaction,0.25374862846174484,0.0084419999999999999,0,-25,25,0.25374862846174484 -coef_adjacent_window_exists_before_this_departure_hour_second_plus_tour_interaction,-0.19834303389910271,-0.059299999999999999,0,-25,25,-0.19834303389910271 -coef_adult_with_children_in_hh_arrive_19_21,0.30942398582506869,0.33600000000000002,0,-25,25,0.30942398582506869 -coef_arrival_constants_am_peak,0.42338231194541198,-1.814822602,0,-25,25,0.42338231194541198 -coef_arrival_constants_early,2.9125415992060972,-0.051990748000000003,0,-25,25,2.9125415992060972 +coef_adjacent_window_exists_after_this_arrival_hour_first_tour_interaction,0.4470328861441506,-0.025700000000000001,0,-25,25,0.4470328861441506 +coef_adjacent_window_exists_after_this_arrival_hour_second_plus_tour_interaction,0.56540266502429914,-0.02734,0,-25,25,0.56540266502429914 +coef_adjacent_window_exists_before_this_departure_hour_first_tour_interaction,1.0259323894447316,0.0084419999999999999,0,-25,25,1.0259323894447316 +coef_adjacent_window_exists_before_this_departure_hour_second_plus_tour_interaction,0.39614317482379957,-0.059299999999999999,0,-25,25,0.39614317482379957 +coef_adult_with_children_in_hh_arrive_19_21,0.31331407963369601,0.33600000000000002,0,-25,25,0.31331407963369601 +coef_arrival_constants_am_peak,0.29950075228772494,-1.814822602,0,-25,25,0.29950075228772494 +coef_arrival_constants_early,2.725267289110473,-0.051990748000000003,0,-25,25,2.725267289110473 coef_arrival_constants_evening,0,0,0,0,0,0 -coef_arrival_constants_late,-1.1910368663347299,-0.86667131500000005,0,-25,25,-1.1910368663347299 -coef_arrival_constants_midday_1,1.4838084605550812,0.00037150099999999999,0,-25,25,1.4838084605550812 -coef_arrival_constants_midday_2,1.5740135366172114,0.53211603100000004,0,-25,25,1.5740135366172114 -coef_arrival_constants_pm_peak_1,1.4096316362715051,0.62848156700000002,0,-25,25,1.4096316362715051 -coef_arrival_constants_pm_peak_2,1.6015853090922501,0.65052141600000002,0,-25,25,1.6015853090922501 -coef_arrival_constants_pm_peak_3,1.1104118303036004,0.40289440599999998,0,-25,25,1.1104118303036004 -coef_arrival_constants_pm_peak_4,0.71897054480267941,0.154213293,0,-25,25,0.71897054480267941 -coef_departure_constants_am_peak_1,-0.46298108436506397,-0.65416357300000005,0,-25,25,-0.46298108436506397 -coef_departure_constants_am_peak_2,1.2593758274992846,0.554282571,0,-25,25,1.2593758274992846 -coef_departure_constants_am_peak_3,1.4967305972755718,1.0505610869999999,0,-25,25,1.4967305972755718 -coef_departure_constants_am_peak_4,1.5743607357021454,0.97156822799999998,0,-25,25,1.5743607357021454 -coef_departure_constants_early,-1.0295111494115279,-1.7401356610000001,0,-25,25,-1.0295111494115279 -coef_departure_constants_evening,-1.9057735658380073,-1.856475096,0,-25,25,-1.9057735658380073 -coef_departure_constants_late,-5.8630966842838257,-8.2288801409999994,0,-25,25,-5.8630966842838257 -coef_departure_constants_midday_1,1.4060318633333218,0.88199198599999995,0,-25,25,1.4060318633333218 -coef_departure_constants_midday_2,0.67297573507452191,0.411103634,0,-25,25,0.67297573507452191 +coef_arrival_constants_late,-1.1446323020070934,-0.86667131500000005,0,-25,25,-1.1446323020070934 +coef_arrival_constants_midday_1,1.3878018540712422,0.00037150099999999999,0,-25,25,1.3878018540712422 +coef_arrival_constants_midday_2,1.5044255895039795,0.53211603100000004,0,-25,25,1.5044255895039795 +coef_arrival_constants_pm_peak_1,1.3575212925702953,0.62848156700000002,0,-25,25,1.3575212925702953 +coef_arrival_constants_pm_peak_2,1.5579462028570845,0.65052141600000002,0,-25,25,1.5579462028570845 +coef_arrival_constants_pm_peak_3,1.0806181769202525,0.40289440599999998,0,-25,25,1.0806181769202525 +coef_arrival_constants_pm_peak_4,0.69735759210419612,0.154213293,0,-25,25,0.69735759210419612 +coef_departure_constants_am_peak_1,-0.52735012006812276,-0.65416357300000005,0,-25,25,-0.52735012006812276 +coef_departure_constants_am_peak_2,1.1919565496484956,0.554282571,0,-25,25,1.1919565496484956 +coef_departure_constants_am_peak_3,1.4522880996476126,1.0505610869999999,0,-25,25,1.4522880996476126 +coef_departure_constants_am_peak_4,1.5114204226663379,0.97156822799999998,0,-25,25,1.5114204226663379 +coef_departure_constants_early,-0.92104869293940328,-1.7401356610000001,0,-25,25,-0.92104869293940328 +coef_departure_constants_evening,-1.892558890126933,-1.856475096,0,-25,25,-1.892558890126933 +coef_departure_constants_late,-5.6265216713840269,-8.2288801409999994,0,-25,25,-5.6265216713840269 +coef_departure_constants_midday_1,1.3642337556324493,0.88199198599999995,0,-25,25,1.3642337556324493 +coef_departure_constants_midday_2,0.64556751877721019,0.411103634,0,-25,25,0.64556751877721019 coef_departure_constants_pm_peak,0,0,0,0,0,0 -coef_destination_in_cbd_duration_shift_effects,0.1161201042988334,0.1067,0,-25,25,0.1161201042988334 -coef_discretionary_tour_duration_lt_2_hours,-0.2639348382984561,-0.69740000000000002,0,-25,25,-0.2639348382984561 +coef_destination_in_cbd_duration_shift_effects,0.11545735776937631,0.1067,0,-25,25,0.11545735776937631 +coef_discretionary_tour_duration_lt_2_hours,-0.28570443543294716,-0.69740000000000002,0,-25,25,-0.28570443543294716 coef_duration_constants_0_to_1_hours,0,0,0,0,0,0 -coef_duration_constants_11_to_13_hours,-0.87732408480084789,-0.955635554,0,-25,25,-0.87732408480084789 -coef_duration_constants_14_to_18_hours,-2.8212978244929676,-1.042580879,0,-25,25,-2.8212978244929676 -coef_duration_constants_2_to_3_hours,0.43539369018467328,0.051385565000000001,0,-25,25,0.43539369018467328 -coef_duration_constants_4_to_5_hours,-0.22360076510836863,-0.59395132100000003,0,-25,25,-0.22360076510836863 -coef_duration_constants_6_to_7_hours,-0.58405912079172817,-0.95115532800000002,0,-25,25,-0.58405912079172817 -coef_duration_constants_8_to_10_hours,-0.52790593216985049,-0.828108399,0,-25,25,-0.52790593216985049 -coef_eat_out_tour_departure_shift_effects,0.10565687284340203,0.075490000000000002,0,-25,25,0.10565687284340203 +coef_duration_constants_11_to_13_hours,-1.0074136237311473,-0.955635554,0,-25,25,-1.0074136237311473 +coef_duration_constants_14_to_18_hours,-2.9449650044604194,-1.042580879,0,-25,25,-2.9449650044604194 +coef_duration_constants_2_to_3_hours,0.3987726313758776,0.051385565000000001,0,-25,25,0.3987726313758776 +coef_duration_constants_4_to_5_hours,-0.27952507767146284,-0.59395132100000003,0,-25,25,-0.27952507767146284 +coef_duration_constants_6_to_7_hours,-0.65619904538586438,-0.95115532800000002,0,-25,25,-0.65619904538586438 +coef_duration_constants_8_to_10_hours,-0.62326492676484069,-0.828108399,0,-25,25,-0.62326492676484069 +coef_eat_out_tour_departure_shift_effects,0.10408922995281709,0.075490000000000002,0,-25,25,0.10408922995281709 coef_escort_tour_arrival_constants_am_peak,0,0,0,0,0,0 coef_escort_tour_arrival_constants_early,0,0,0,0,0,0 -coef_escort_tour_arrival_constants_evening,-0.87885259097970236,-0.53691872799999996,0,-25,25,-0.87885259097970236 -coef_escort_tour_arrival_constants_late,-1.2544150933448788,-1.008290213,0,-25,25,-1.2544150933448788 +coef_escort_tour_arrival_constants_evening,-0.8931637699400693,-0.53691872799999996,0,-25,25,-0.8931637699400693 +coef_escort_tour_arrival_constants_late,-1.1770797692787565,-1.008290213,0,-25,25,-1.1770797692787565 coef_escort_tour_arrival_constants_midday_1,0,0,0,0,0,0 coef_escort_tour_arrival_constants_midday_2,0,0,0,0,0,0 coef_escort_tour_arrival_constants_pm_peak_1,0,0,0,0,0,0 coef_escort_tour_arrival_constants_pm_peak_2,0,0,0,0,0,0 coef_escort_tour_arrival_constants_pm_peak_3,0,0,0,0,0,0 coef_escort_tour_arrival_constants_pm_peak_4,0,0,0,0,0,0 -coef_escort_tour_departure_constants_am_peak_1,0.70430916557367373,-1.1123577529999999,0,-25,25,0.70430916557367373 -coef_escort_tour_departure_constants_am_peak_2,2.276345350741189,0.69878818499999995,0,-25,25,2.276345350741189 -coef_escort_tour_departure_constants_am_peak_3,2.3008963449265081,1.1962688130000001,0,-25,25,2.3008963449265081 -coef_escort_tour_departure_constants_am_peak_4,0.49313656586014942,-0.22525822100000001,0,-25,25,0.49313656586014942 -coef_escort_tour_departure_constants_early,-0.49391001353581915,-1.7401356610000001,0,-25,25,-0.49391001353581915 -coef_escort_tour_departure_constants_evening,-4.8022685133416925,-3.9487328110000002,0,-25,25,-4.8022685133416925 -coef_escort_tour_departure_constants_late,-6.8574861910304179,-8.2288801409999994,0,-25,25,-6.8574861910304179 -coef_escort_tour_departure_constants_midday_1,0.7464465452279021,0.028662017000000001,0,-25,25,0.7464465452279021 +coef_escort_tour_departure_constants_am_peak_1,0.48291466233476604,-1.1123577529999999,0,-25,25,0.48291466233476604 +coef_escort_tour_departure_constants_am_peak_2,2.0132225505169186,0.69878818499999995,0,-25,25,2.0132225505169186 +coef_escort_tour_departure_constants_am_peak_3,2.1327074544894873,1.1962688130000001,0,-25,25,2.1327074544894873 +coef_escort_tour_departure_constants_am_peak_4,0.29911257358855436,-0.22525822100000001,0,-25,25,0.29911257358855436 +coef_escort_tour_departure_constants_early,-0.5151665525123571,-1.7401356610000001,0,-25,25,-0.5151665525123571 +coef_escort_tour_departure_constants_evening,-4.7638859598327503,-3.9487328110000002,0,-25,25,-4.7638859598327503 +coef_escort_tour_departure_constants_late,-6.5766274559402813,-8.2288801409999994,0,-25,25,-6.5766274559402813 +coef_escort_tour_departure_constants_midday_1,0.64080123578544568,0.028662017000000001,0,-25,25,0.64080123578544568 coef_escort_tour_departure_constants_midday_2,0,0,0,0,0,0 -coef_escort_tour_departure_constants_pm_peak,-1.8233595967851086,-1.180140161,0,-25,25,-1.8233595967851086 +coef_escort_tour_departure_constants_pm_peak,-1.78947739154685,-1.180140161,0,-25,25,-1.78947739154685 coef_escort_tour_duration_constants_0_to_1_hours,0,0,0,0,0,0 -coef_escort_tour_duration_constants_11_to_13_hours,-6.3931476306146751,-2.9743649759999999,0,-25,25,-6.3931476306146751 -coef_escort_tour_duration_constants_14_to_18_hours,-9.4197927932485896,-2.5074471460000001,0,-25,25,-9.4197927932485896 -coef_escort_tour_duration_constants_2_to_3_hours,-2.5955922813145555,-2.0420138969999999,0,-25,25,-2.5955922813145555 -coef_escort_tour_duration_constants_4_to_5_hours,-3.5228888691144959,-2.880293896,0,-25,25,-3.5228888691144959 -coef_escort_tour_duration_constants_6_to_7_hours,-3.979277237854689,-2.9735337309999998,0,-25,25,-3.979277237854689 -coef_escort_tour_duration_constants_8_to_10_hours,-5.1583159232705968,-3.0202137580000001,0,-25,25,-5.1583159232705968 -coef_first_of_2_plus_tours_for_same_purpose_departure_shift_effect,-0.28520075435325948,-0.2364,0,-25,25,-0.28520075435325948 -coef_free_flow_round_trip_auto_time_shift_effects_duration,0.0049919355433122436,0.0047410000000000004,0,-25,25,0.0049919355433122436 -coef_maintenance_tour_depart_before_7,0.041996701210282046,-0.88260000000000005,0,-25,25,0.041996701210282046 -coef_maintenance_tour_departure_shift_effects,-0.036635545903427789,-0.1489,0,-25,25,-0.036635545903427789 -coef_maintenance_tour_duration_shift_effects,-0.1159437577045862,-0.083720000000000003,0,-25,25,-0.1159437577045862 -coef_number_of_escort_tours_departure_shift_effects,0.1152135976407198,0.020129999999999999,0,-25,25,0.1152135976407198 -coef_number_of_individual_non_mandatory_tours_excluding_escort_departure_shift_effects,0.10437475708930999,0.038960000000000002,0,-25,25,0.10437475708930999 -coef_number_of_joint_tours_departure_shift_effects,0.12192044424883711,0.052080000000000001,0,-25,25,0.12192044424883711 -coef_number_of_mandatory_tours_departure_shift_effects,0.17081676715336708,0.046730000000000001,0,-25,25,0.17081676715336708 -coef_ratio_of_individual_non_mandatory_tours_to_be_scheduled_to_number_of_unscheduled_hours,5.6436190430109816,-13.630000000000001,0,-25,25,5.6436190430109816 -coef_school_child_age_16_plus_departure_shift_effects,0.00028295279835436216,0.072660000000000002,0,-25,25,0.00028295279835436216 -coef_school_child_age_16_plus_duration_shift_effects,0.30612186264250524,0.20949999999999999,0,-25,25,0.30612186264250524 -coef_school_child_age_under_16_departure_shift_effects,0.11079613464663726,0.04657,0,-25,25,0.11079613464663726 -coef_school_child_age_under_16_duration_shift_effects,0.33821236944890215,0.32719999999999999,0,-25,25,0.33821236944890215 -coef_school_child_under_16_arrive_after_22,-0.40240965942253298,-1.1799999999999999,0,-25,25,-0.40240965942253298 -coef_shopping_tour_arrive_after_22,-2.5452262719424956,-0.60270000000000001,0,-25,25,-2.5452262719424956 -coef_shopping_tour_depart_before_8,-0.78710037865743798,-1.0369999999999999,0,-25,25,-0.78710037865743798 -coef_shopping_tour_departure_shift_effects,0.015067331790855299,-0.060150000000000002,0,-25,25,0.015067331790855299 -coef_shopping_tour_duration_lt_2_hours,0.94341972947847441,0.51680000000000004,0,-25,25,0.94341972947847441 -coef_shopping_tour_duration_shift_effects,-0.075331047793831374,-0.1208,0,-25,25,-0.075331047793831374 -coef_some_previously_scheduled_tour_begins_in_this_arrival_hour,-0.05438519391868582,-0.3992,0,-25,25,-0.05438519391868582 -coef_some_previously_scheduled_tour_ends_in_this_departure_hour,-0.34572651304702828,-0.45619999999999999,0,-25,25,-0.34572651304702828 -coef_subsequent_of_2_plus_tours_for_same_purpose_duration_shift_effect,-0.11510735547695737,-0.1731,0,-25,25,-0.11510735547695737 +coef_escort_tour_duration_constants_11_to_13_hours,-6.4203324936004007,-2.9743649759999999,0,-25,25,-6.4203324936004007 +coef_escort_tour_duration_constants_14_to_18_hours,-9.2410499900666672,-2.5074471460000001,0,-25,25,-9.2410499900666672 +coef_escort_tour_duration_constants_2_to_3_hours,-2.600430326093385,-2.0420138969999999,0,-25,25,-2.600430326093385 +coef_escort_tour_duration_constants_4_to_5_hours,-3.5279045037998431,-2.880293896,0,-25,25,-3.5279045037998431 +coef_escort_tour_duration_constants_6_to_7_hours,-3.9930263301758195,-2.9735337309999998,0,-25,25,-3.9930263301758195 +coef_escort_tour_duration_constants_8_to_10_hours,-5.1822459061747068,-3.0202137580000001,0,-25,25,-5.1822459061747068 +coef_first_of_2_plus_tours_for_same_purpose_departure_shift_effect,-0.30486234400858059,-0.2364,0,-25,25,-0.30486234400858059 +coef_free_flow_round_trip_auto_time_shift_effects_duration,0.0050429319781780269,0.0047410000000000004,0,-25,25,0.0050429319781780269 +coef_maintenance_tour_depart_before_7,0.052350894248540496,-0.88260000000000005,0,-25,25,0.052350894248540496 +coef_maintenance_tour_departure_shift_effects,-0.041432645424667555,-0.1489,0,-25,25,-0.041432645424667555 +coef_maintenance_tour_duration_shift_effects,-0.11717847942290449,-0.083720000000000003,0,-25,25,-0.11717847942290449 +coef_number_of_escort_tours_departure_shift_effects,0.1003621581289256,0.020129999999999999,0,-25,25,0.1003621581289256 +coef_number_of_individual_non_mandatory_tours_excluding_escort_departure_shift_effects,0.098215723941853808,0.038960000000000002,0,-25,25,0.098215723941853808 +coef_number_of_joint_tours_departure_shift_effects,0.12273335177942903,0.052080000000000001,0,-25,25,0.12273335177942903 +coef_number_of_mandatory_tours_departure_shift_effects,0.1726876041064854,0.046730000000000001,0,-25,25,0.1726876041064854 +coef_ratio_of_individual_non_mandatory_tours_to_be_scheduled_to_number_of_unscheduled_hours,5.7502384461866223,-13.630000000000001,0,-25,25,5.7502384461866223 +coef_school_child_age_16_plus_departure_shift_effects,0.002334001767561128,0.072660000000000002,0,-25,25,0.002334001767561128 +coef_school_child_age_16_plus_duration_shift_effects,0.30512330091593426,0.20949999999999999,0,-25,25,0.30512330091593426 +coef_school_child_age_under_16_departure_shift_effects,0.1028104094868405,0.04657,0,-25,25,0.1028104094868405 +coef_school_child_age_under_16_duration_shift_effects,0.33667040040989843,0.32719999999999999,0,-25,25,0.33667040040989843 +coef_school_child_under_16_arrive_after_22,-0.29870062575080597,-1.1799999999999999,0,-25,25,-0.29870062575080597 +coef_shopping_tour_arrive_after_22,-2.4414941640980703,-0.60270000000000001,0,-25,25,-2.4414941640980703 +coef_shopping_tour_depart_before_8,-0.84790437236499061,-1.0369999999999999,0,-25,25,-0.84790437236499061 +coef_shopping_tour_departure_shift_effects,0.0090862416686240035,-0.060150000000000002,0,-25,25,0.0090862416686240035 +coef_shopping_tour_duration_lt_2_hours,0.912982413595951,0.51680000000000004,0,-25,25,0.912982413595951 +coef_shopping_tour_duration_shift_effects,-0.08012681553606732,-0.1208,0,-25,25,-0.08012681553606732 +coef_some_previously_scheduled_tour_begins_in_this_arrival_hour,-0.004765610611209883,-0.3992,0,-25,25,-0.004765610611209883 +coef_some_previously_scheduled_tour_ends_in_this_departure_hour,-0.28646699211356075,-0.45619999999999999,0,-25,25,-0.28646699211356075 +coef_subsequent_of_2_plus_tours_for_same_purpose_duration_shift_effect,-0.10477927540016083,-0.1731,0,-25,25,-0.10477927540016083 coef_subsequent_tour_must_start_after_previous_tour_for_this_purpose_ends,-999,-999,0,-999,-999,-999 -coef_university_student_arrive_after_22,-0.53729852771430953,0.54659999999999997,0,-25,25,-0.53729852771430953 -coef_visit_tour_departure_shift_effects,0.16959832172498893,0.096879999999999994,0,-25,25,0.16959832172498893 -coef_visit_tour_duration_shift_effects,0.15578338984481574,0.1638,0,-25,25,0.15578338984481574 +coef_university_student_arrive_after_22,-0.4840281170137713,0.54659999999999997,0,-25,25,-0.4840281170137713 +coef_visit_tour_departure_shift_effects,0.1674492151765343,0.096879999999999994,0,-25,25,0.1674492151765343 +coef_visit_tour_duration_shift_effects,0.15890840399796446,0.1638,0,-25,25,0.15890840399796446 diff --git a/activitysim/examples/__init__.py b/activitysim/examples/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/activitysim/examples/example_manifest.yaml b/activitysim/examples/example_manifest.yaml index 5c29a9924c..eee74e9fcd 100644 --- a/activitysim/examples/example_manifest.yaml +++ b/activitysim/examples/example_manifest.yaml @@ -81,7 +81,7 @@ include: - prototype_mtc/configs - prototype_mtc/configs_mp - - prototype_mtc/data + # prototype_mtc/data # load data from activitysim_resources - prototype_mtc/output - https://media.githubusercontent.com/media/activitysim/activitysim_resources/master/mtc_data_sf/skims.omx data/skims.omx @@ -652,7 +652,7 @@ - placeholder_sandag/../prototype_mtc/configs prototype_mtc - placeholder_sandag/configs_1_zone - - placeholder_sandag/data_1 + # placeholder_sandag/data_1 # load data from activitysim_resources instead - placeholder_sandag/output_1 - https://media.githubusercontent.com/media/activitysim/activitysim_resources/master/sandag_1_zone_data_full/households.csv data_1/households.csv @@ -708,7 +708,7 @@ - placeholder_sandag/../placeholder_psrc/configs placeholder_psrc - placeholder_sandag/configs_2_zone - - placeholder_sandag/data_2 + # placeholder_sandag/data_2 # load data from activitysim_resources instead - placeholder_sandag/output_2 - https://media.githubusercontent.com/media/activitysim/activitysim_resources/master/sandag_2_zone_data_full/households.csv data_2/households.csv @@ -771,12 +771,14 @@ # activitysim run -c configs_3_zone -c prototype_mtc/configs -d data_3 -o output_3 -s settings_mp.yaml # cd .. include: - - placeholder_sandag/data_3 + # placeholder_sandag/data_3 # load data from activitysim_resources instead - placeholder_sandag/../prototype_mtc/configs prototype_mtc - placeholder_sandag/configs_3_zone - placeholder_sandag/configs_skip_accessibility - placeholder_sandag/output_3 + - placeholder_sandag/data_3/cached_accessibility.csv.gz + data_3/cached_accessibility.csv.gz - https://media.githubusercontent.com/media/activitysim/activitysim_resources/master/sandag_3_zone_data_full/taz_skims1.omx data_3/taz_skims1.omx 5b56d0e79ec671e37f8c71f7fedd741d7bf32d2bced866ab1f03f3973fccce8c @@ -912,4 +914,3 @@ - prototype_mwcog/output - prototype_mwcog/README.MD - prototype_mwcog/simulation.py - \ No newline at end of file diff --git a/activitysim/examples/optimize_example_data.py b/activitysim/examples/optimize_example_data.py new file mode 100644 index 0000000000..490bea110c --- /dev/null +++ b/activitysim/examples/optimize_example_data.py @@ -0,0 +1,41 @@ +import os + +import openmatrix +import pandas as pd + + +def patch_example_sandag_1_zone(example_dir): + + cwd = os.getcwd() + try: + os.chdir(example_dir) + skims = openmatrix.open_file("data_1/skims1.omx", mode="a") + skims_lookup = skims.root["lookup"] + + zone_name = None + if len(skims_lookup._v_children) == 1: + zone_name = list(skims_lookup._v_children)[0] + zone_data = skims_lookup[zone_name] + rezone = pd.Series( + pd.RangeIndex(1, zone_data.shape[0] + 1), + index=zone_data[:], + ) + else: + rezone = None + + if rezone is not None: + households = pd.read_csv("data_1/households.csv") + households["TAZ"] = households["TAZ"].map(rezone) + households.to_csv("data_1/households.csv", index=False) + + land_use = pd.read_csv("data_1/land_use.csv") + land_use["TAZ"] = land_use["TAZ"].map(rezone) + land_use.to_csv("data_1/land_use.csv", index=False) + + if zone_name: + skims_lookup[zone_name]._f_remove() + + skims.close() + + finally: + os.chdir(cwd) diff --git a/activitysim/examples/placeholder_multiple_zone/configs_2_zone/settings.yaml b/activitysim/examples/placeholder_multiple_zone/configs_2_zone/settings.yaml index a775eaf685..68675f3021 100644 --- a/activitysim/examples/placeholder_multiple_zone/configs_2_zone/settings.yaml +++ b/activitysim/examples/placeholder_multiple_zone/configs_2_zone/settings.yaml @@ -64,6 +64,11 @@ input_table_list: - TOPOLOGY - TERMINAL - access_dist_transit + - tablename: land_use_taz + filename: taz.csv + index_col: TAZ + recode_columns: + TAZ: zero-based #resume_after: initialize_landuse @@ -82,3 +87,5 @@ output_tables: # trace origin, destination in accessibility calculation; comment out or leave empty for no trace trace_od: [5000, 11000] + +recode_pipeline_columns: False diff --git a/activitysim/examples/placeholder_multiple_zone/configs_3_zone/settings.yaml b/activitysim/examples/placeholder_multiple_zone/configs_3_zone/settings.yaml index 0cde54e8ec..245a1b66c3 100644 --- a/activitysim/examples/placeholder_multiple_zone/configs_3_zone/settings.yaml +++ b/activitysim/examples/placeholder_multiple_zone/configs_3_zone/settings.yaml @@ -83,3 +83,5 @@ output_tables: # trace origin, destination in accessibility calculation; comment out or leave empty for no trace #trace_od: [5000, 11000] + +recode_pipeline_columns: False diff --git a/activitysim/examples/placeholder_multiple_zone/test/regress/final_tours_2_zone.csv b/activitysim/examples/placeholder_multiple_zone/test/regress/final_tours_2_zone.csv index 38d480be5f..64da1ba53d 100644 --- a/activitysim/examples/placeholder_multiple_zone/test/regress/final_tours_2_zone.csv +++ b/activitysim/examples/placeholder_multiple_zone/test/regress/final_tours_2_zone.csv @@ -26,7 +26,7 @@ tour_id,person_id,tour_type,tour_type_count,tour_type_num,tour_num,tour_count,to 58514946,1427193,shopping,1,1,1,1,non_mandatory,1,16000.0,25000.0,703381,130.0,13.0,19.0,6.0,,12.523150856688828,WALK,-1.1307531749443571,,,0out_0in,shopping 58514982,1427194,othmaint,1,1,1,2,non_mandatory,1,25000.0,25000.0,703381,43.0,7.0,13.0,6.0,,13.741081970110171,WALK,0.7102432373197154,,,0out_0in,othmaint 58514990,1427194,social,1,1,2,2,non_mandatory,1,11000.0,25000.0,703381,160.0,16.0,22.0,6.0,,13.587548572119903,WALK,-0.3434452146802574,,,1out_2in,social -64479052,1572659,shopping,1,1,1,1,non_mandatory,1,19000.0,6000.0,763879,50.0,7.0,20.0,13.0,,12.695032127873429,TNC_SINGLE,-1.5395149128365382,,,3out_0in,shopping +64479052,1572659,shopping,1,1,1,1,non_mandatory,1,19000.0,6000.0,763879,50.0,7.0,20.0,13.0,,12.695032127873429,TNC_SINGLE,-1.5395149128365384,,,3out_0in,shopping 64490158,1572930,othmaint,1,1,2,2,non_mandatory,1,14000.0,9000.0,764150,57.0,8.0,11.0,3.0,,13.876979292356584,WALK_LRF,-0.8284966847360843,,,1out_1in,othmaint 64490163,1572930,shopping,1,1,1,2,non_mandatory,1,21000.0,9000.0,764150,104.0,11.0,16.0,5.0,,12.652319439420722,WALK,-0.7117743765170423,,,0out_0in,shopping 66923525,1632281,eat,1,1,1,1,atwork,1,5000.0,14000.0,823501,125.0,13.0,14.0,1.0,,15.582337201253203,WALK,5.627962154350834,,66923560.0,1out_0in,atwork @@ -36,7 +36,7 @@ tour_id,person_id,tour_type,tour_type_count,tour_type_num,tour_num,tour_count,to 76904608,1875722,eatout,1,1,1,1,non_mandatory,1,17000.0,16000.0,982875,91.0,10.0,16.0,6.0,,14.206952836182584,WALK,2.455435658701909,,,1out_0in,eatout 88521376,2159057,work,1,1,1,1,mandatory,1,21000.0,20000.0,1099626,48.0,7.0,18.0,11.0,,,WALK,0.012760765917479,no_subtours,,0out_0in,work 88521409,2159058,school,1,1,1,1,mandatory,1,10000.0,20000.0,1099626,145.0,15.0,15.0,0.0,,,WALK_LOC,0.7098750900924855,,,0out_0in,univ -88521450,2159059,school,1,1,1,1,mandatory,1,11000.0,20000.0,1099626,58.0,8.0,12.0,4.0,,,WALK,-0.6251066468993006,,,0out_0in,school +88521450,2159059,school,1,1,1,1,mandatory,1,11000.0,20000.0,1099626,58.0,8.0,12.0,4.0,,,WALK,-0.6251066468993004,,,0out_0in,school 100798519,2458502,shopping,1,1,1,1,joint,2,5000.0,8000.0,1173905,54.0,8.0,8.0,0.0,children,12.849118307199914,SHARED2FREE,-0.5111781667673684,,,0out_0in,shopping 100798528,2458500,othmaint,1,1,1,1,non_mandatory,1,9000.0,8000.0,1173905,120.0,12.0,20.0,8.0,,13.933008315199084,WALK,-0.0534146023738154,,,0out_0in,othmaint 100798550,2458501,escort,1,1,1,1,non_mandatory,1,17000.0,8000.0,1173905,146.0,15.0,16.0,1.0,,12.452436194889506,WALK_LRF,-2.030352537291469,,,0out_0in,escort @@ -51,14 +51,14 @@ tour_id,person_id,tour_type,tour_type_count,tour_type_num,tour_num,tour_count,to 120410774,2936848,eatout,1,1,2,2,non_mandatory,1,6000.0,11000.0,1286557,136.0,14.0,15.0,1.0,,13.385908233280974,DRIVEALONEFREE,0.5446813508509882,,,0out_0in,eatout 120410793,2936848,othdiscr,1,1,1,2,non_mandatory,1,25000.0,11000.0,1286557,146.0,15.0,16.0,1.0,,13.998407599705123,DRIVEALONEFREE,0.0531249030428947,,,1out_0in,othdiscr 125537687,3061894,shopping,1,1,1,1,non_mandatory,1,5000.0,24000.0,1363467,113.0,12.0,13.0,1.0,,12.67844496957057,WALK,0.1794050478758298,,,0out_0in,shopping -125537720,3061895,othdiscr,1,1,2,2,non_mandatory,1,9000.0,24000.0,1363467,164.0,17.0,19.0,2.0,,13.855717382920384,WALK_HVY,0.1180104930269069,,,0out_0in,othdiscr +125537720,3061895,othdiscr,1,1,2,2,non_mandatory,1,9000.0,24000.0,1363467,164.0,17.0,19.0,2.0,,13.855717382920384,WALK_HVY,0.1180104930269067,,,0out_0in,othdiscr 125537723,3061895,othmaint,1,1,1,2,non_mandatory,1,4000.0,24000.0,1363467,146.0,15.0,16.0,1.0,,13.843895836691548,WALK,0.4346074611532325,,,0out_0in,othmaint 125537734,3061895,work,1,1,1,1,mandatory,1,2000.0,24000.0,1363467,26.0,6.0,13.0,7.0,,,WALK,0.4289367938120317,no_subtours,,0out_0in,work 130727777,3188483,othdiscr,1,1,1,1,joint,3,3000.0,25000.0,1402945,70.0,9.0,9.0,0.0,adults,14.490562913707262,WALK,-2.2237959104289464,,,0out_0in,othdiscr 130727801,3188482,work,1,1,1,1,mandatory,1,14000.0,25000.0,1402945,63.0,8.0,17.0,9.0,,,WALK,1.470809103053866,no_subtours,,1out_0in,work 130727842,3188483,work,1,1,1,1,mandatory,1,15000.0,25000.0,1402945,77.0,9.0,16.0,7.0,,,WALK_LOC,1.476215281092868,no_subtours,,0out_0in,work -130727875,3188484,school,1,1,2,2,mandatory,1,12000.0,25000.0,1402945,154.0,16.0,16.0,0.0,,,WALK,3.3072495217751663,,,0out_1in,univ -130727883,3188484,work,1,1,1,2,mandatory,1,5000.0,25000.0,1402945,71.0,9.0,10.0,1.0,,,WALK_LOC,2.1088774790757463,no_subtours,,0out_0in,work +130727875,3188484,school,1,1,2,2,mandatory,1,12000.0,25000.0,1402945,155.0,16.0,17.0,1.0,,,WALK,3.303113744185396,,,0out_2in,univ +130727883,3188484,work,1,1,1,2,mandatory,1,5000.0,25000.0,1402945,73.0,9.0,12.0,3.0,,,WALK_LOC,2.0839353031691825,no_subtours,,0out_0in,work 130727924,3188485,work,1,1,1,1,mandatory,1,1000.0,25000.0,1402945,96.0,10.0,21.0,11.0,,,WALK_LOC,1.7666619623619302,no_subtours,,1out_1in,work 132571948,3233462,eatout,1,1,3,3,non_mandatory,1,13000.0,17000.0,1445222,146.0,15.0,16.0,1.0,,15.090110400259128,WALK,3.6847853902920167,,,0out_0in,eatout 132571975,3233462,shopping,2,1,1,3,non_mandatory,1,16000.0,17000.0,1445222,162.0,17.0,17.0,0.0,,14.059936928241624,WALK_LRF,2.5837938479671587,,,0out_0in,shopping @@ -111,5 +111,5 @@ tour_id,person_id,tour_type,tour_type_count,tour_type_num,tour_num,tour_count,to 308013110,7512514,social,1,1,3,3,non_mandatory,1,16000.0,8000.0,2821179,154.0,16.0,16.0,0.0,,13.794538448328227,WALK_LOC,0.0836844432714275,,,0out_0in,social 308050751,7513432,work,1,1,1,1,mandatory,1,9000.0,8000.0,2822097,10.0,5.0,15.0,10.0,,,BIKE,0.1541226783048151,no_subtours,,0out_0in,work 308055753,7513554,work,1,1,1,1,mandatory,1,7000.0,8000.0,2822219,11.0,5.0,16.0,11.0,,,WALK,5.8674311173207885,no_subtours,,0out_0in,work -308056204,7513565,work,1,1,1,1,mandatory,1,12000.0,8000.0,2822230,82.0,9.0,21.0,12.0,,,WALK,5.818260825925703,no_subtours,,1out_1in,work +308056204,7513565,work,1,1,1,1,mandatory,1,12000.0,8000.0,2822230,82.0,9.0,21.0,12.0,,,WALK,5.818260825925704,no_subtours,,1out_1in,work 308464222,7523517,othdiscr,1,1,1,1,non_mandatory,1,16000.0,7000.0,2832182,59.0,8.0,13.0,5.0,,14.941709019837395,WALK_LOC,1.725989964213542,,,0out_0in,othdiscr diff --git a/activitysim/examples/placeholder_multiple_zone/test/regress/final_trips_2_zone.csv b/activitysim/examples/placeholder_multiple_zone/test/regress/final_trips_2_zone.csv index a26e48e7bb..8a555ba053 100644 --- a/activitysim/examples/placeholder_multiple_zone/test/regress/final_trips_2_zone.csv +++ b/activitysim/examples/placeholder_multiple_zone/test/regress/final_trips_2_zone.csv @@ -17,7 +17,7 @@ trip_id,person_id,household_id,primary_purpose,trip_num,outbound,trip_count,dest 36757261,112064,112064,shopping,1,False,1,16000,11000,4594657,home,,17,WALK,3.9642911737176 86627409,264107,226869,work,1,True,1,15000,9000,10828426,work,,7,WALK_LRF,1.451342899763582 86627413,264107,226869,work,1,False,1,9000,15000,10828426,home,,19,WALK_LRF,1.4433154732713112 -86627625,264108,226869,othdiscr,1,True,1,2000,9000,10828453,othdiscr,,10,WALK_LRF,0.8013218089436901 +86627625,264108,226869,othdiscr,1,True,1,2000,9000,10828453,othdiscr,,10,WALK_LRF,0.8013218089436903 86627629,264108,226869,othdiscr,1,False,2,25000,2000,10828453,escort,32.83572582954258,11,WALK,0.1335656823355492 86627630,264108,226869,othdiscr,2,False,2,9000,25000,10828453,home,,11,WALK_LOC,12.099030970556203 106170305,323689,256660,work,1,True,1,12000,10000,13271288,work,,11,WALK_LOC,3.790246557613184 @@ -51,8 +51,8 @@ trip_id,person_id,household_id,primary_purpose,trip_num,outbound,trip_count,dest 211388445,644476,386761,work,1,False,1,16000,12000,26423555,home,,17,WALK,3.516861371044758 211388721,644477,386761,shopping,1,True,1,12000,16000,26423590,shopping,,14,WALK,3.1625662738073435 211388725,644477,386761,shopping,1,False,1,16000,12000,26423590,home,,14,WALK,3.162369002002967 -211389033,644478,386761,school,1,True,1,22000,16000,26423629,school,,8,WALK_LOC,1.1439151605607298 -211389037,644478,386761,school,1,False,1,16000,22000,26423629,home,,16,WALK,1.2205853035954894 +211389033,644478,386761,school,1,True,1,22000,16000,26423629,school,,8,WALK_LOC,1.1439151605607296 +211389037,644478,386761,school,1,False,1,16000,22000,26423629,home,,16,WALK,1.2205853035954892 415214745,1265898,568785,othdiscr,1,True,1,9000,17000,51901843,othdiscr,,9,WALK_LRF,10.8917991771908 415214749,1265898,568785,othdiscr,1,False,1,17000,9000,51901843,home,,20,WALK_LRF,11.199024497487024 468119569,1427193,703381,shopping,1,True,1,16000,25000,58514946,shopping,,13,WALK,5.874499232376766 @@ -107,7 +107,7 @@ trip_id,person_id,household_id,primary_purpose,trip_num,outbound,trip_count,dest 841877257,2566698,1196298,work,1,True,1,4000,25000,105234657,work,,6,WALK,1.0198269854172115 841877261,2566698,1196298,work,1,False,1,25000,4000,105234657,home,,18,WALK,0.9937585802563552 841877849,2566700,1196298,school,1,True,1,5000,25000,105234731,school,,7,WALK_LOC,4.047398168186926 -841877853,2566700,1196298,school,1,False,1,25000,5000,105234731,home,,15,WALK_LOC,3.9643518377422713 +841877853,2566700,1196298,school,1,False,1,25000,5000,105234731,home,,15,WALK_LOC,3.964351837742272 841878177,2566701,1196298,school,1,True,1,8000,25000,105234772,school,,8,WALK_LOC,11.36280438733663 841878181,2566701,1196298,school,1,False,1,25000,8000,105234772,home,,13,WALK_LOC,10.81430067240017 841878505,2566702,1196298,school,1,True,1,25000,25000,105234813,school,,12,WALK,12.82461586997922 @@ -137,10 +137,11 @@ trip_id,person_id,household_id,primary_purpose,trip_num,outbound,trip_count,dest 1045822737,3188483,1402945,work,1,True,1,15000,25000,130727842,work,,9,WALK,1.1621136398928025 1045822741,3188483,1402945,work,1,False,1,25000,15000,130727842,home,,16,WALK_LOC,1.0332661932426703 1045823001,3188484,1402945,univ,1,True,1,12000,25000,130727875,univ,,16,WALK,3.8010257036427793 -1045823005,3188484,1402945,univ,1,False,2,25000,12000,130727875,shopping,40.59787148529891,16,WALK,3.725146070305983 -1045823006,3188484,1402945,univ,2,False,2,25000,25000,130727875,home,,16,WALK,12.824615870159588 +1045823005,3188484,1402945,univ,1,False,3,25000,12000,130727875,shopping,40.597871512333725,17,WALK,3.725146077579001 +1045823006,3188484,1402945,univ,2,False,3,7000,25000,130727875,othmaint,56.60102011468401,17,WALK,12.03329580904219 +1045823007,3188484,1402945,univ,3,False,3,25000,7000,130727875,home,,17,WALK,12.646537396714637 1045823065,3188484,1402945,work,1,True,1,5000,25000,130727883,work,,9,WALK,4.08199251781282 -1045823069,3188484,1402945,work,1,False,1,25000,5000,130727883,home,,10,WALK_LOC,3.9154781479629737 +1045823069,3188484,1402945,work,1,False,1,25000,5000,130727883,home,,12,WALK_LOC,3.90954672164702 1045823393,3188485,1402945,work,1,True,2,7000,25000,130727924,escort,30.71610667465545,10,WALK_LOC,11.388230172922947 1045823394,3188485,1402945,work,2,True,2,1000,7000,130727924,work,,12,WALK,-0.1910319526120294 1045823397,3188485,1402945,work,1,False,2,7000,1000,130727924,eatout,30.600694971829377,20,WALK_LOC,-0.1403015748111612 @@ -229,8 +230,8 @@ trip_id,person_id,household_id,primary_purpose,trip_num,outbound,trip_count,dest 1767186578,5387763,2223027,work,2,True,2,5000,7000,220898322,work,,8,WALK,4.031252242930563 1767186581,5387763,2223027,work,1,False,2,8000,5000,220898322,othmaint,36.964562786529065,17,WALK,3.9594482660927697 1767186582,5387763,2223027,work,2,False,2,9000,8000,220898322,home,,17,WALK,10.001230529962584 -1767666161,5389226,2223759,atwork,1,True,1,13000,21000,220958270,atwork,,13,WALK,-0.8469767119587287 -1767666165,5389226,2223759,atwork,1,False,1,21000,13000,220958270,work,,13,WALK,-0.8469846953017646 +1767666161,5389226,2223759,atwork,1,True,1,13000,21000,220958270,atwork,,13,WALK,-0.8469767119587291 +1767666165,5389226,2223759,atwork,1,False,1,21000,13000,220958270,work,,13,WALK,-0.8469846953017649 1767666233,5389226,2223759,eatout,1,True,1,16000,16000,220958279,eatout,,20,WALK,6.618624772185435 1767666237,5389226,2223759,eatout,1,False,1,16000,16000,220958279,home,,20,WALK,6.618624772185435 1767666441,5389226,2223759,work,1,True,1,21000,16000,220958305,work,,9,WALK,2.520654632402534 @@ -254,7 +255,7 @@ trip_id,person_id,household_id,primary_purpose,trip_num,outbound,trip_count,dest 2464104645,7512514,2821179,eatout,1,False,1,8000,19000,308013080,home,,16,WALK,-1.474310526482455 2464104857,7512514,2821179,shopping,1,True,1,6000,8000,308013107,shopping,,17,WALK,12.612248978887928 2464104861,7512514,2821179,shopping,1,False,1,8000,6000,308013107,home,,19,WALK,12.322088998148194 -2464104881,7512514,2821179,social,1,True,1,16000,8000,308013110,social,,16,WALK_LOC,3.494452156419123 +2464104881,7512514,2821179,social,1,True,1,16000,8000,308013110,social,,16,WALK_LOC,3.4944521564191238 2464104885,7512514,2821179,social,1,False,1,8000,16000,308013110,home,,16,WALK_LOC,3.4381818382545646 2464406009,7513432,2822097,work,1,True,1,9000,8000,308050751,work,,5,BIKE,8.0614192494138 2464406013,7513432,2822097,work,1,False,1,8000,9000,308050751,home,,15,BIKE,8.061419249304613 @@ -264,5 +265,5 @@ trip_id,person_id,household_id,primary_purpose,trip_num,outbound,trip_count,dest 2464449634,7513565,2822230,work,2,True,2,12000,9000,308056204,work,,9,WALK,2.6765203030859817 2464449637,7513565,2822230,work,1,False,2,6000,12000,308056204,escort,33.34376957078302,12,WALK,3.1913129432158387 2464449638,7513565,2822230,work,2,False,2,8000,6000,308056204,home,,21,WALK,9.3863428519612 -2467713777,7523517,2832182,othdiscr,1,True,1,16000,7000,308464222,othdiscr,,8,WALK,6.612046694416055 +2467713777,7523517,2832182,othdiscr,1,True,1,16000,7000,308464222,othdiscr,,8,WALK,6.612046694416056 2467713781,7523517,2832182,othdiscr,1,False,1,7000,16000,308464222,home,,13,WALK_LOC,6.681639068176178 diff --git a/activitysim/examples/placeholder_multiple_zone/test/test_multiple_zone.py b/activitysim/examples/placeholder_multiple_zone/test/test_multiple_zone.py index c0f19d3b6f..40d4a41a06 100644 --- a/activitysim/examples/placeholder_multiple_zone/test/test_multiple_zone.py +++ b/activitysim/examples/placeholder_multiple_zone/test/test_multiple_zone.py @@ -2,6 +2,7 @@ # See full license in LICENSE.txt. import os import subprocess +import sys import pandas as pd import pandas.testing as pdt @@ -28,12 +29,12 @@ def mtc_example_path(dirname): def build_data(): if os.environ.get("TRAVIS") != "true": - subprocess.check_call( - ["coverage", "run", example_path("scripts/two_zone_example_data.py")] - ) - subprocess.check_call( - ["coverage", "run", example_path("scripts/three_zone_example_data.py")] - ) + if os.environ.get("GITHUB_ACTIONS") == "true": + go = ["coverage", "run"] + else: + go = [sys.executable] + subprocess.check_call(go + [example_path("scripts/two_zone_example_data.py")]) + subprocess.check_call(go + [example_path("scripts/three_zone_example_data.py")]) @pytest.fixture(scope="module") @@ -89,7 +90,10 @@ def regress(zone): elif zone == "3": run_args = run_args + ["-s", "settings_static"] - subprocess.run(["coverage", "run", "-a", file_path] + run_args, check=True) + if os.environ.get("GITHUB_ACTIONS") == "true": + subprocess.run(["coverage", "run", "-a", file_path] + run_args, check=True) + else: + subprocess.run([sys.executable, file_path] + run_args, check=True) regress(zone) diff --git a/activitysim/examples/placeholder_psrc/configs/logging.yaml b/activitysim/examples/placeholder_psrc/configs/logging.yaml index 71ac15cc1f..46838a7a2a 100755 --- a/activitysim/examples/placeholder_psrc/configs/logging.yaml +++ b/activitysim/examples/placeholder_psrc/configs/logging.yaml @@ -1,54 +1,70 @@ -# Config for logging -# ------------------ -# See http://docs.python.org/2.7/library/logging.config.html#configuration-dictionary-schema - -logging: - version: 1 - disable_existing_loggers: true - - - # Configuring the default (root) logger is highly recommended - root: - level: NOTSET - handlers: [console, logfile] - - loggers: - - activitysim: - level: INFO - handlers: [console, logfile] - propagate: false - - orca: - level: WARN - handlers: [console, logfile] - propagate: false - - handlers: - - logfile: - class: logging.FileHandler - filename: !!python/object/apply:activitysim.core.config.log_file_path ['activitysim.log'] - mode: w - formatter: fileFormatter - level: NOTSET - - console: - class: logging.StreamHandler - stream: ext://sys.stdout - formatter: simpleFormatter - level: NOTSET - - formatters: - - simpleFormatter: - class: logging.Formatter - # format: '%(levelname)s - %(name)s - %(message)s' - format: '%(levelname)s - %(message)s' - datefmt: '%d/%m/%Y %H:%M:%S' - - fileFormatter: - class: logging.Formatter - format: '%(asctime)s - %(levelname)s - %(name)s - %(message)s' - datefmt: '%d/%m/%Y %H:%M:%S' - +# Config for logging +# ------------------ +# See http://docs.python.org/2.7/library/logging.config.html#configuration-dictionary-schema + +logging: + version: 1 + disable_existing_loggers: true + + + # Configuring the default (root) logger is highly recommended + root: + level: NOTSET + handlers: [console, logfile] + + loggers: + + activitysim: + level: INFO + handlers: [console, logfile] + propagate: false + + orca: + level: WARN + handlers: [console, logfile] + propagate: false + + filelock: + level: WARN + + sharrow: + level: INFO + + blib2to3: + level: WARN + + black: + level: WARN + + handlers: + + logfile: + class: logging.FileHandler + filename: !!python/object/apply:activitysim.core.config.log_file_path ['activitysim.log'] + mode: w + formatter: fileFormatter + level: NOTSET + + console: + class: logging.StreamHandler + stream: ext://sys.stdout + formatter: elapsedFormatter + level: NOTSET + + formatters: + + simpleFormatter: + class: logging.Formatter + # format: '%(levelname)s - %(name)s - %(message)s' + format: '%(levelname)s - %(message)s' + datefmt: '%d/%m/%Y %H:%M:%S' + + fileFormatter: + class: logging.Formatter + format: '%(asctime)s - %(levelname)s - %(name)s - %(message)s' + datefmt: '%d/%m/%Y %H:%M:%S' + + elapsedFormatter: + (): activitysim.core.tracing.ElapsedTimeFormatter + format: '[{elapsedTime}] {levelname:s}: {message:s}' + style: '{' diff --git a/activitysim/examples/placeholder_psrc/configs/mandatory_tour_frequency.csv b/activitysim/examples/placeholder_psrc/configs/mandatory_tour_frequency.csv index 848bbf77aa..51094ee822 100755 --- a/activitysim/examples/placeholder_psrc/configs/mandatory_tour_frequency.csv +++ b/activitysim/examples/placeholder_psrc/configs/mandatory_tour_frequency.csv @@ -25,10 +25,10 @@ util_can_walk_to_work_retired,Can walk to work - Retired interaction,(ptype == 5 util_can_walk_to_school_univ,Can walk to school - University student interaction,(ptype == 3) & (distance_to_school < 3),,,,coef_can_walk_to_work_school2, util_can_walk_to_school_driving_age_child,Can walk to school - Driving-age child interaction,(ptype == 6) & (distance_to_school < 3),,,,coef_can_walk_to_work_school2, util_can_walk_to_school_pre_driving_age_child,Can walk to school - Pre-driving age child who is in school interaction,(ptype == 7) & (distance_to_school < 3),,,,coef_can_walk_to_work_school2, -util_can_walk_to_work_or_school_ft,Can walk to work or school - Full-time worker interaction,(ptype == 1) & (distance_to_work < 3 | distance_to_school < 3),,,,,coef_can_walk_to_work_and_school -util_can_walk_to_work_or_school_pt,Can walk to work or school - Part-time worker interaction,(ptype == 2) & (distance_to_work < 3 | distance_to_school < 3),,,,,coef_can_walk_to_work_and_school -util_can_walk_to_work_or_school_univ,Can walk to work or school - University student interaction,(ptype == 3) & (distance_to_work < 3 | distance_to_school < 3),,,,,coef_can_walk_to_work_and_school -util_can_walk_to_work_or_school_driving_age_child,Can walk to work or school - Driving-age child interaction,(ptype == 6) & (distance_to_work < 3 | distance_to_school < 3),,,,,coef_can_walk_to_work_and_school +util_can_walk_to_work_or_school_ft,Can walk to work or school - Full-time worker interaction,(ptype == 1) & ((distance_to_work < 3) | (distance_to_school < 3)),,,,,coef_can_walk_to_work_and_school +util_can_walk_to_work_or_school_pt,Can walk to work or school - Part-time worker interaction,(ptype == 2) & ((distance_to_work < 3) | (distance_to_school < 3)),,,,,coef_can_walk_to_work_and_school +util_can_walk_to_work_or_school_univ,Can walk to work or school - University student interaction,(ptype == 3) & ((distance_to_work < 3) | (distance_to_school < 3)),,,,,coef_can_walk_to_work_and_school +util_can_walk_to_work_or_school_driving_age_child,Can walk to work or school - Driving-age child interaction,(ptype == 6) & ((distance_to_work < 3) | (distance_to_school < 3)),,,,,coef_can_walk_to_work_and_school util_round_trip_auto_time_to_work_ft,Round trip auto time to work - Full-time worker interaction,(ptype == 1) * roundtrip_auto_time_to_work,,coef_round_trip_auto_time_to_work_work2,,,coef_round_trip_auto_time_to_work_school2 util_round_trip_auto_time_to_work_pt,Round trip auto time to work - Part-time worker interaction,(ptype == 2) * roundtrip_auto_time_to_work,,coef_round_trip_auto_time_to_work_work2,,,coef_round_trip_auto_time_to_work_school2 util_round_trip_auto_time_to_work_univ,Round trip auto time to work - University student interaction,(ptype == 3) * roundtrip_auto_time_to_work,,coef_round_trip_auto_time_to_work_work2,,,coef_round_trip_auto_time_to_work_school2 diff --git a/activitysim/examples/placeholder_psrc/configs/settings.yaml b/activitysim/examples/placeholder_psrc/configs/settings.yaml index 779d6a1822..c8f4214378 100755 --- a/activitysim/examples/placeholder_psrc/configs/settings.yaml +++ b/activitysim/examples/placeholder_psrc/configs/settings.yaml @@ -1,176 +1,185 @@ -#inherit_settings: True - -# activitysim run -c configs -d data -o output - - -# number of households to simulate -households_sample_size: 100 -# simulate all households -# households_sample_size: 0 - -chunk_size: 0 - -# assume enough RAM to not chunk -chunk_training_mode: disabled - -# set false to disable variability check in simple_simulate and interaction_simulate -check_for_variability: False - -# - shadow pricing global switches - -# turn shadow_pricing on and off for all models (e.g. school and work) -# shadow pricing is deprecated for less than full samples -# see shadow_pricing.yaml for additional settings -use_shadow_pricing: False - -# turn writing of sample_tables on and off for all models -# (if True, tables will be written if DEST_CHOICE_SAMPLE_TABLE_NAME is specified in individual model settings) -want_dest_choice_sample_tables: False - -# - tracing - -# trace household id; comment out or leave empty for no trace -# households with all tour types -trace_hh_id: - -# trace origin, destination in accessibility calculation; comment out or leave empty for no trace -trace_od: - - -# input tables -input_table_list: - - tablename: households - filename: households.csv - index_col: household_id - rename_columns: - HHID: household_id - PERSONS: hhsize - workers: num_workers - VEHICL: auto_ownership - MAZ: home_zone_id - keep_columns: - - home_zone_id - - income - - hhsize - - HHT - - auto_ownership - - num_workers - - tablename: persons - filename: persons.csv - index_col: person_id - rename_columns: - PERID: person_id - keep_columns: - - household_id - - age - - PNUM - - sex - - pemploy - - pstudent - - ptype - - tablename: land_use - filename: land_use.csv - index_col: zone_id - rename_columns: - MAZ: zone_id - COUNTY: county_id - keep_columns: - - TAZ - - DISTRICT - - SD - - county_id - - TOTHH - - TOTPOP - - TOTACRE - - RESACRE - - CIACRE - - TOTEMP - - AGE0519 - - RETEMPN - - FPSEMPN - - HEREMPN - - OTHEMPN - - AGREMPN - - MWTEMPN - - PRKCST - - OPRKCST - - area_type - - HSENROLL - - COLLFTE - - COLLPTE - - TOPOLOGY - - TERMINAL -# - access_dist_transit - -# to resume after last successful checkpoint, specify resume_after: _ -#resume_after: trip_scheduling - -models: - - initialize_landuse - - initialize_households - - compute_accessibility - - school_location - - workplace_location - - auto_ownership_simulate - - free_parking - - cdap_simulate - - mandatory_tour_frequency - - mandatory_tour_scheduling - - joint_tour_frequency - - joint_tour_composition - - joint_tour_participation - - joint_tour_destination - - joint_tour_scheduling - - non_mandatory_tour_frequency - - non_mandatory_tour_destination - - non_mandatory_tour_scheduling - - tour_mode_choice_simulate - - atwork_subtour_frequency - - atwork_subtour_destination - - atwork_subtour_scheduling - - atwork_subtour_mode_choice - - stop_frequency - - trip_purpose - - trip_destination - - trip_purpose_and_destination - - trip_scheduling - - trip_mode_choice - - write_data_dictionary - - track_skim_usage - - write_trip_matrices - - write_tables - - -output_tables: - h5_store: False - action: include - prefix: final_ - sort: True - tables: - - checkpoints - - accessibility - - land_use - - households - - persons - - tours - - trips - - joint_tour_participants - -# area_types less than this are considered urban -urban_threshold: 4 -cbd_threshold: 2 -rural_threshold: 6 - - -# value_of_time = lognormal(np.log(median_value_of_time * mu), sigma).clip(min_vot, max_vot) - -min_value_of_time: 1 -max_value_of_time: 50 -distributed_vot_mu: 0.684 -distributed_vot_sigma: 0.85 - -household_median_value_of_time: - 1: 6.01 - 2: 8.81 - 3: 10.44 - 4: 12.86 +#inherit_settings: True + +# activitysim run -c configs -d data -o output + + +# number of households to simulate +households_sample_size: 100 +# simulate all households +# households_sample_size: 0 + +chunk_size: 0 + +# assume enough RAM to not chunk +chunk_training_mode: disabled + +# set false to disable variability check in simple_simulate and interaction_simulate +check_for_variability: False + +# - shadow pricing global switches + +# turn shadow_pricing on and off for all models (e.g. school and work) +# shadow pricing is deprecated for less than full samples +# see shadow_pricing.yaml for additional settings +use_shadow_pricing: False + +# turn writing of sample_tables on and off for all models +# (if True, tables will be written if DEST_CHOICE_SAMPLE_TABLE_NAME is specified in individual model settings) +want_dest_choice_sample_tables: False + +# - tracing + +# trace household id; comment out or leave empty for no trace +# households with all tour types +trace_hh_id: + +# trace origin, destination in accessibility calculation; comment out or leave empty for no trace +trace_od: + + +# input tables +input_table_list: + - tablename: households + filename: households.csv + index_col: household_id + rename_columns: + HHID: household_id + PERSONS: hhsize + workers: num_workers + VEHICL: auto_ownership + MAZ: home_zone_id + recode_columns: + home_zone_id: land_use.zone_id + keep_columns: + - home_zone_id + - income + - hhsize + - HHT + - auto_ownership + - num_workers + - tablename: persons + filename: persons.csv + index_col: person_id + rename_columns: + PERID: person_id + keep_columns: + - household_id + - age + - PNUM + - sex + - pemploy + - pstudent + - ptype + - tablename: land_use + filename: land_use.csv + index_col: zone_id + rename_columns: + MAZ: zone_id + COUNTY: county_id + recode_columns: + zone_id: zero-based + keep_columns: + - TAZ + - DISTRICT + - SD + - county_id + - TOTHH + - TOTPOP + - TOTACRE + - RESACRE + - CIACRE + - TOTEMP + - AGE0519 + - RETEMPN + - FPSEMPN + - HEREMPN + - OTHEMPN + - AGREMPN + - MWTEMPN + - PRKCST + - OPRKCST + - area_type + - HSENROLL + - COLLFTE + - COLLPTE + - TOPOLOGY + - TERMINAL +# - access_dist_transit + - tablename: land_use_taz + filename: taz.csv + index_col: TAZ + recode_columns: + TAZ: zero-based + +# to resume after last successful checkpoint, specify resume_after: _ +#resume_after: trip_scheduling + +models: + - initialize_landuse + - initialize_households + - compute_accessibility + - school_location + - workplace_location + - auto_ownership_simulate + - free_parking + - cdap_simulate + - mandatory_tour_frequency + - mandatory_tour_scheduling + - joint_tour_frequency + - joint_tour_composition + - joint_tour_participation + - joint_tour_destination + - joint_tour_scheduling + - non_mandatory_tour_frequency + - non_mandatory_tour_destination + - non_mandatory_tour_scheduling + - tour_mode_choice_simulate + - atwork_subtour_frequency + - atwork_subtour_destination + - atwork_subtour_scheduling + - atwork_subtour_mode_choice + - stop_frequency + - trip_purpose + - trip_destination + - trip_purpose_and_destination + - trip_scheduling + - trip_mode_choice + - write_data_dictionary + - track_skim_usage + - write_trip_matrices + - write_tables + + +output_tables: + h5_store: False + action: include + prefix: final_ + sort: True + tables: + - checkpoints + - accessibility + - land_use + - households + - persons + - tours + - trips + - joint_tour_participants + +# area_types less than this are considered urban +urban_threshold: 4 +cbd_threshold: 2 +rural_threshold: 6 + + +# value_of_time = lognormal(np.log(median_value_of_time * mu), sigma).clip(min_vot, max_vot) + +min_value_of_time: 1 +max_value_of_time: 50 +distributed_vot_mu: 0.684 +distributed_vot_sigma: 0.85 + +household_median_value_of_time: + 1: 6.01 + 2: 8.81 + 3: 10.44 + 4: 12.86 diff --git a/activitysim/examples/placeholder_psrc/configs/trip_destination.csv b/activitysim/examples/placeholder_psrc/configs/trip_destination.csv index 99434ae2cc..ab15ba6889 100644 --- a/activitysim/examples/placeholder_psrc/configs/trip_destination.csv +++ b/activitysim/examples/placeholder_psrc/configs/trip_destination.csv @@ -1,8 +1,8 @@ Description,Expression,work,univ,school,escort,shopping,eatout,othmaint,social,othdiscr,atwork # size term,"@np.log1p(size_terms.get(df.dest_taz, df.purpose))",1,1,1,1,1,1,1,1,1,1 # no attractions,"@size_terms.get(df.dest_taz, df.purpose) == 0",-999,-999,-999,-999,-999,-999,-999,-999,-999,-999 -size term,"@np.log1p(size_terms.get(df.alt_dest, df.purpose))",1,1,1,1,1,1,1,1,1,1 -no attractions,"@size_terms.get(df.alt_dest, df.purpose) == 0",-999,-999,-999,-999,-999,-999,-999,-999,-999,-999 +size term,"@np.log1p(size_terms.get(df.alt_dest, df.purpose)) # sharrow: np.log1p(size_terms['arry'])",1,1,1,1,1,1,1,1,1,1 +no attractions,"@size_terms.get(df.alt_dest, df.purpose) == 0 # sharrow: size_terms['arry'] == 0",-999,-999,-999,-999,-999,-999,-999,-999,-999,-999 #stop zone CBD area type,"@reindex(land_use.area_type, df.dest_taz) < setting('cbd_threshold')",,,,,,,,,, distance (calibration adjustment individual - inbound),@(~df.is_joint & ~df.outbound) * (od_skims['DIST'] + dp_skims['DIST']),-0.04972591574229,-0.0613,-0.1056,-0.1491,-0.1192,-0.1029,-0.0962,-0.1329,-0.126172224,-0.122334597 distance (calibration adjustment individual - outbound),@(~df.is_joint & df.outbound) * (od_skims['DIST'] + dp_skims['DIST']),0.147813278663948,-0.0613,-0.1056,-0.1491,-0.1192,-0.1029,-0.0962,-0.1329,-0.126172224,-0.122334597 diff --git a/activitysim/examples/placeholder_psrc/configs/trip_destination_annotate_trips_preprocessor.csv b/activitysim/examples/placeholder_psrc/configs/trip_destination_annotate_trips_preprocessor.csv index 1a1afb0748..9f2d502d3a 100755 --- a/activitysim/examples/placeholder_psrc/configs/trip_destination_annotate_trips_preprocessor.csv +++ b/activitysim/examples/placeholder_psrc/configs/trip_destination_annotate_trips_preprocessor.csv @@ -7,4 +7,7 @@ Description,Target,Expression #,,not needed as school is not chosen as an intermediate trip destination #,_grade_school,"(df.primary_purpose == 'school') & reindex(persons.is_gradeschool, df.person_id)" #,size_segment,"df.primary_purpose.where(df.primary_purpose != 'school', np.where(_grade_school,'gradeschool', 'highschool'))" -,tour_leg_dest,"np.where(df.outbound,reindex(tours.destination, df.tour_id), reindex(tours.origin, df.tour_id))" \ No newline at end of file +,purpose_index_num,"size_terms.get_cols(df.purpose)" +,tour_mode_is_walk,"reindex(tours.tour_mode, df.tour_id)=='WALK'" +,tour_mode_is_bike,"reindex(tours.tour_mode, df.tour_id)=='BIKE'" +,tour_leg_dest,"np.where(df.outbound,reindex(tours.destination, df.tour_id), reindex(tours.origin, df.tour_id)).astype(int)" diff --git a/activitysim/examples/placeholder_psrc/configs/trip_destination_sample.csv b/activitysim/examples/placeholder_psrc/configs/trip_destination_sample.csv index ecf943d7d8..92817c63fb 100644 --- a/activitysim/examples/placeholder_psrc/configs/trip_destination_sample.csv +++ b/activitysim/examples/placeholder_psrc/configs/trip_destination_sample.csv @@ -8,8 +8,8 @@ Not available if bike tour not within biking distance,@(df.tour_mode=='BIKE') & #If transit tour is not in walk sub-zone it must be walkable,,,,,,,,,,, # size term,"@np.log1p(size_terms.get(df.dest_taz, df.purpose))",1,1,1,1,1,1,1,1,1,1 # no attractions,"@size_terms.get(df.dest_taz, df.purpose) == 0",-999,-999,-999,-999,-999,-999,-999,-999,-999,-999 -size term,"@np.log1p(size_terms.get(df.alt_dest, df.purpose))",1,1,1,1,1,1,1,1,1,1 -no attractions,"@size_terms.get(df.alt_dest, df.purpose) == 0",-999,-999,-999,-999,-999,-999,-999,-999,-999,-999 +size term,"@np.log1p(size_terms.get(df.alt_dest, df.purpose)) # sharrow: np.log1p(size_terms['arry'])",1,1,1,1,1,1,1,1,1,1 +no attractions,"@size_terms.get(df.alt_dest, df.purpose) == 0 # sharrow: size_terms['arry'] == 0",-999,-999,-999,-999,-999,-999,-999,-999,-999,-999 #stop zone CBD area type,"@reindex(land_use.area_type, df.dest_taz) < setting('cbd_threshold')",,,,,,,,,, distance (calibration adjustment individual - inbound),@(~df.is_joint & ~df.outbound) * (_od_DIST + _dp_DIST),-0.04972591574229,-0.0613,-0.1056,-0.1491,-0.1192,-0.1029,-0.0962,-0.1329,-0.126172224,-0.122334597 distance (calibration adjustment individual - outbound),@(~df.is_joint & df.outbound) * (_od_DIST + _dp_DIST),0.147813278663948,-0.0613,-0.1056,-0.1491,-0.1192,-0.1029,-0.0962,-0.1329,-0.126172224,-0.122334597 diff --git a/activitysim/examples/placeholder_psrc/test/configs/settings.yaml b/activitysim/examples/placeholder_psrc/test/configs/settings.yaml index 12820a3412..e01bf1e83d 100644 --- a/activitysim/examples/placeholder_psrc/test/configs/settings.yaml +++ b/activitysim/examples/placeholder_psrc/test/configs/settings.yaml @@ -37,3 +37,5 @@ output_tables: sort: True tables: - trips + +recode_pipeline_columns: False diff --git a/activitysim/examples/placeholder_sandag/configs_1_zone/logging.yaml b/activitysim/examples/placeholder_sandag/configs_1_zone/logging.yaml new file mode 100644 index 0000000000..13e533abba --- /dev/null +++ b/activitysim/examples/placeholder_sandag/configs_1_zone/logging.yaml @@ -0,0 +1,69 @@ +# Config for logging +# ------------------ +# See http://docs.python.org/2.7/library/logging.config.html#configuration-dictionary-schema + +logging: + version: 1 + disable_existing_loggers: true + + + # Configuring the default (root) logger is highly recommended + root: + level: NOTSET + handlers: [console, logfile] + + loggers: + + activitysim: + level: INFO + handlers: [console, logfile] + propagate: false + + orca: + level: WARN + handlers: [console, logfile] + propagate: false + + filelock: + level: WARN + + sharrow: + level: INFO + + blib2to3: + level: WARN + + black: + level: WARN + + handlers: + + logfile: + class: logging.FileHandler + filename: !!python/object/apply:activitysim.core.config.log_file_path ['activitysim.log'] + mode: w + formatter: fileFormatter + level: NOTSET + + console: + class: logging.StreamHandler + stream: ext://sys.stdout + level: !!python/object/apply:activitysim.core.mp_tasks.if_sub_task [WARNING, NOTSET] + formatter: elapsedFormatter + + formatters: + + simpleFormatter: + class: logging.Formatter + format: '%(levelname)s - %(message)s' + datefmt: '%d/%m/%Y %H:%M:%S' + + fileFormatter: + class: logging.Formatter + format: '%(asctime)s - %(levelname)s - %(name)s - %(message)s' + datefmt: '%d/%m/%Y %H:%M:%S' + + elapsedFormatter: + (): activitysim.core.tracing.ElapsedTimeFormatter + format: '[{elapsedTime}] {levelname:s}: {message:s}' + style: '{' diff --git a/activitysim/examples/placeholder_sandag/configs_1_zone/network_los.yaml b/activitysim/examples/placeholder_sandag/configs_1_zone/network_los.yaml index 8560d7d2f4..d1a9b9d4e6 100644 --- a/activitysim/examples/placeholder_sandag/configs_1_zone/network_los.yaml +++ b/activitysim/examples/placeholder_sandag/configs_1_zone/network_los.yaml @@ -8,7 +8,25 @@ write_skim_cache: False zone_system: 1 -taz_skims: skims*.omx +taz_skims: + omx: skims*.omx + zarr: skims.zarr + zarr-digital-encoding: + - {regex: .*_BOARDS , joint_dict: true} + - {regex: .*_DDIST , joint_dict: true} + - {regex: .*_DTIM , joint_dict: true} + - {regex: .*_FAR , joint_dict: true} + - {regex: .*_IWAIT , joint_dict: true} + - {regex: .*_KEYIVT , joint_dict: true} + - {regex: .*_TOTIVT , joint_dict: true} + - {regex: .*_FERRYIVT , joint_dict: true} + - {regex: .*_WAIT , joint_dict: true} + - {regex: .*_WAUX , joint_dict: true} + - {regex: .*_XWAIT , joint_dict: true} + - {regex: .*_BTOLL , joint_dict: true} + - {regex: .*_DIST , joint_dict: true} + - {regex: .*_TIME , joint_dict: true} + - {regex: .*_VTOLL , joint_dict: true} skim_time_periods: time_window: 1440 diff --git a/activitysim/examples/placeholder_sandag/configs_1_zone/settings.yaml b/activitysim/examples/placeholder_sandag/configs_1_zone/settings.yaml index 4c3317401c..e478e64fa0 100644 --- a/activitysim/examples/placeholder_sandag/configs_1_zone/settings.yaml +++ b/activitysim/examples/placeholder_sandag/configs_1_zone/settings.yaml @@ -28,6 +28,8 @@ input_table_list: workers: num_workers VEHICL: auto_ownership TAZ: home_zone_id + recode_columns: + home_zone_id: land_use.zone_id keep_columns: - home_zone_id - income @@ -61,6 +63,8 @@ input_table_list: rename_columns: TAZ: zone_id # person_id is the required index column COUNTY: county_id + recode_columns: + zone_id: zero-based keep_columns: - DISTRICT - SD @@ -186,3 +190,4 @@ household_median_value_of_time: 2: 8.81 3: 10.44 4: 12.86 + diff --git a/activitysim/examples/placeholder_sandag/configs_2_zone/legacy-1.0.4/logging.yaml b/activitysim/examples/placeholder_sandag/configs_2_zone/legacy-1.0.4/logging.yaml new file mode 100644 index 0000000000..3d62e8064b --- /dev/null +++ b/activitysim/examples/placeholder_sandag/configs_2_zone/legacy-1.0.4/logging.yaml @@ -0,0 +1,54 @@ +# Config for logging +# ------------------ +# See http://docs.python.org/2.7/library/logging.config.html#configuration-dictionary-schema + +logging: + version: 1 + disable_existing_loggers: true + + + # Configuring the default (root) logger is highly recommended + root: + level: NOTSET + handlers: [console, logfile] + + loggers: + + activitysim: + level: INFO + handlers: [console, logfile] + propagate: false + + orca: + level: WARNING + handlers: [console, logfile] + propagate: false + + handlers: + + logfile: + class: logging.FileHandler + filename: !!python/object/apply:activitysim.core.config.log_file_path ['activitysim.log'] + mode: w + formatter: fileFormatter + level: NOTSET + + console: + class: logging.StreamHandler + stream: ext://sys.stdout + formatter: simpleFormatter + level: !!python/object/apply:activitysim.core.mp_tasks.if_sub_task [WARNING, NOTSET] + + formatters: + + simpleFormatter: + class: logging.Formatter + format: !!python/object/apply:activitysim.core.mp_tasks.if_sub_task [ + '%(processName)-10s %(levelname)s - %(name)s - %(message)s', + '%(levelname)s - %(name)s - %(message)s'] + datefmt: '%d/%m/%Y %H:%M:%S' + + fileFormatter: + class: logging.Formatter + format: '%(asctime)s - %(levelname)s - %(name)s - %(message)s' + datefmt: '%d/%m/%Y %H:%M:%S' diff --git a/activitysim/examples/placeholder_sandag/configs_2_zone/legacy-1.0.4/network_los.yaml b/activitysim/examples/placeholder_sandag/configs_2_zone/legacy-1.0.4/network_los.yaml new file mode 100644 index 0000000000..8aee14644f --- /dev/null +++ b/activitysim/examples/placeholder_sandag/configs_2_zone/legacy-1.0.4/network_los.yaml @@ -0,0 +1,34 @@ +#inherit_settings: True + +# read cached skims (using numpy memmap) from output directory (memmap is faster than omx ) +read_skim_cache: False +# write memmapped cached skims to output directory after reading from omx, for use in subsequent runs +write_skim_cache: False + +zone_system: 2 + +# glob 'skims*.omx' will match one or more files: skims.omx, skims1.omx, skims2.omx... +taz_skims: skims*.omx + + +maz: maz.csv + +maz_to_maz: + tables: + - maz_to_maz_walk.csv + - maz_to_maz_bike.csv + + # maz_to_maz blending distance (missing or 0 means no blending) + max_blend_distance: + # blend distance of 0 means no blending + DISTBIKE: 0 + DISTWALK: 0 + + # missing means use the skim value itself rather than DIST skim (e.g. DISTBIKE) + #blend_distance_skim_name: DIST + +skim_time_periods: + time_window: 1440 + period_minutes: 60 + periods: [0, 6, 11, 16, 20, 24] + labels: ['EA', 'AM', 'MD', 'PM', 'EV'] diff --git a/activitysim/examples/placeholder_sandag/configs_2_zone/legacy-1.0.4/settings.yaml b/activitysim/examples/placeholder_sandag/configs_2_zone/legacy-1.0.4/settings.yaml new file mode 100644 index 0000000000..9a047df3d4 --- /dev/null +++ b/activitysim/examples/placeholder_sandag/configs_2_zone/legacy-1.0.4/settings.yaml @@ -0,0 +1,167 @@ +inherit_settings: True + +# number of households to simulate +# simulate all households +households_sample_size: 0 + +# set false to disable variability check in simple_simulate and interaction_simulate +check_for_variability: False + +# presampling +want_dest_choice_presampling: True + +# - shadow pricing global switches + +# turn shadow_pricing on and off for all models (e.g. school and work) +# shadow pricing is deprecated for less than full samples +# see shadow_pricing.yaml for additional settings +use_shadow_pricing: False + +# turn writing of sample_tables on and off for all models +# (if True, tables will be written if DEST_CHOICE_SAMPLE_TABLE_NAME is specified in individual model settings) +want_dest_choice_sample_tables: False + +# - tracing + +# trace household id; comment out or leave empty for no trace +# households with all tour types +trace_hh_id: + +# trace origin, destination in accessibility calculation; comment out or leave empty for no trace +trace_od: + + +# input tables +input_table_list: + - tablename: households + filename: households.csv + index_col: household_id + rename_columns: + HHID: household_id + PERSONS: hhsize + workers: num_workers + VEHICL: auto_ownership + MAZ: home_zone_id + keep_columns: + - home_zone_id + - income + - hhsize + - HHT + - auto_ownership + - num_workers + - tablename: persons + filename: persons.csv + index_col: person_id + rename_columns: + PERID: person_id + keep_columns: + - household_id + - age + - PNUM + - sex + - pemploy + - pstudent + - ptype + - tablename: land_use + filename: land_use.csv + index_col: zone_id + rename_columns: + MAZ: zone_id + COUNTY: county_id + keep_columns: + - TAZ + - DISTRICT + - SD + - county_id + - TOTHH + - TOTPOP + - TOTACRE + - RESACRE + - CIACRE + - TOTEMP + - AGE0519 + - RETEMPN + - FPSEMPN + - HEREMPN + - OTHEMPN + - AGREMPN + - MWTEMPN + - PRKCST + - OPRKCST + - area_type + - HSENROLL + - COLLFTE + - COLLPTE + - TOPOLOGY + - TERMINAL +# - access_dist_transit + +models: + - initialize_landuse + - initialize_households + - compute_accessibility + - school_location + - workplace_location + - auto_ownership_simulate + - free_parking + - cdap_simulate + - mandatory_tour_frequency + - mandatory_tour_scheduling + - joint_tour_frequency + - joint_tour_composition + - joint_tour_participation + - joint_tour_destination + - joint_tour_scheduling + - non_mandatory_tour_frequency + - non_mandatory_tour_destination + - non_mandatory_tour_scheduling + - tour_mode_choice_simulate + - atwork_subtour_frequency + - atwork_subtour_destination + - atwork_subtour_scheduling + - atwork_subtour_mode_choice + - stop_frequency + - trip_purpose + - trip_destination + - trip_purpose_and_destination + - trip_scheduling + - trip_mode_choice + - write_data_dictionary + - track_skim_usage + - write_trip_matrices + - write_tables + + +output_tables: + h5_store: False + action: include + prefix: final_ + sort: True + tables: + - checkpoints + - accessibility + - land_use + - households + - persons + - tours + - trips + - joint_tour_participants + +# area_types less than this are considered urban +urban_threshold: 4 +cbd_threshold: 2 +rural_threshold: 6 + + +# value_of_time = lognormal(np.log(median_value_of_time * mu), sigma).clip(min_vot, max_vot) + +min_value_of_time: 1 +max_value_of_time: 50 +distributed_vot_mu: 0.684 +distributed_vot_sigma: 0.85 + +household_median_value_of_time: + 1: 6.01 + 2: 8.81 + 3: 10.44 + 4: 12.86 diff --git a/activitysim/examples/placeholder_sandag/configs_2_zone/legacy-1.0.4/settings_mp.yaml b/activitysim/examples/placeholder_sandag/configs_2_zone/legacy-1.0.4/settings_mp.yaml new file mode 100644 index 0000000000..5002c6284a --- /dev/null +++ b/activitysim/examples/placeholder_sandag/configs_2_zone/legacy-1.0.4/settings_mp.yaml @@ -0,0 +1,47 @@ +inherit_settings: settings.yaml + +# turn shadow_pricing on and off for all models (e.g. school and work) +# shadow pricing is deprecated for less than full samples +# see shadow_pricing.yaml for additional settings +use_shadow_pricing: False + +households_sample_size: 0 +multiprocess: True +num_processes: 28 + +# raise error if any sub-process fails without waiting for others to complete +fail_fast: True + +# presampling +want_dest_choice_presampling: True + +multiprocess_steps: + - name: mp_initialize + begin: initialize_landuse + - name: mp_accessibility + begin: compute_accessibility + slice: + tables: + - accessibility + - name: mp_households + begin: school_location + slice: + tables: + - households + - persons + - name: mp_summarize + begin: write_data_dictionary + +output_tables: + h5_store: False + action: include + prefix: final_ + sort: True + tables: + - checkpoints + - accessibility + - land_use + - households + - persons + - tours + - trips diff --git a/activitysim/examples/placeholder_sandag/configs_2_zone/legacy-1.0.4/trip_destination_annotate_trips_preprocessor.csv b/activitysim/examples/placeholder_sandag/configs_2_zone/legacy-1.0.4/trip_destination_annotate_trips_preprocessor.csv new file mode 100644 index 0000000000..9db6a951da --- /dev/null +++ b/activitysim/examples/placeholder_sandag/configs_2_zone/legacy-1.0.4/trip_destination_annotate_trips_preprocessor.csv @@ -0,0 +1,12 @@ +Description,Target,Expression +#,, +,tour_mode,"reindex(tours.tour_mode, df.tour_id)" +,_tod,"np.where(df.outbound,reindex_i(tours.start, df.tour_id),reindex_i(tours.end, df.tour_id))" +,trip_period,network_los.skim_time_period_label(_tod) +,is_joint,"reindex(tours.tour_category, df.tour_id)=='joint'" +#,,not needed as school is not chosen as an intermediate trip destination +#,_grade_school,"(df.primary_purpose == 'school') & reindex(persons.is_gradeschool, df.person_id)" +#,size_segment,"df.primary_purpose.where(df.primary_purpose != 'school', np.where(_grade_school,'gradeschool', 'highschool'))" +,tour_mode_is_walk,"reindex(tours.tour_mode, df.tour_id)=='WALK'" +,tour_mode_is_bike,"reindex(tours.tour_mode, df.tour_id)=='BIKE'" +,tour_leg_dest,"np.where(df.outbound,reindex(tours.destination, df.tour_id), reindex(tours.origin, df.tour_id)).astype(int)" diff --git a/activitysim/examples/placeholder_sandag/configs_2_zone/network_los.yaml b/activitysim/examples/placeholder_sandag/configs_2_zone/network_los.yaml index 8aee14644f..600c542fdd 100644 --- a/activitysim/examples/placeholder_sandag/configs_2_zone/network_los.yaml +++ b/activitysim/examples/placeholder_sandag/configs_2_zone/network_los.yaml @@ -8,7 +8,25 @@ write_skim_cache: False zone_system: 2 # glob 'skims*.omx' will match one or more files: skims.omx, skims1.omx, skims2.omx... -taz_skims: skims*.omx +taz_skims: + omx: skims*.omx + zarr: skims.zarr + zarr-digital-encoding: + - {regex: .*_BOARDS , joint_dict: true} + - {regex: .*_DDIST , joint_dict: true} + - {regex: .*_DTIM , joint_dict: true} + - {regex: .*_FAR , joint_dict: true} + - {regex: .*_IWAIT , joint_dict: true} + - {regex: .*_KEYIVT , joint_dict: true} + - {regex: .*_TOTIVT , joint_dict: true} + - {regex: .*_FERRYIVT , joint_dict: true} + - {regex: .*_WAIT , joint_dict: true} + - {regex: .*_WAUX , joint_dict: true} + - {regex: .*_XWAIT , joint_dict: true} + - {regex: .*_BTOLL , joint_dict: true} + - {regex: .*_TIME , joint_dict: true} + - {regex: .*_VTOLL , joint_dict: true} + - {regex: "^(?!_s_).*_DIST" , joint_dict: true} maz: maz.csv diff --git a/activitysim/examples/placeholder_sandag/configs_2_zone/settings.yaml b/activitysim/examples/placeholder_sandag/configs_2_zone/settings.yaml index 9a047df3d4..2d341f552a 100644 --- a/activitysim/examples/placeholder_sandag/configs_2_zone/settings.yaml +++ b/activitysim/examples/placeholder_sandag/configs_2_zone/settings.yaml @@ -42,6 +42,8 @@ input_table_list: workers: num_workers VEHICL: auto_ownership MAZ: home_zone_id + recode_columns: + home_zone_id: land_use.zone_id keep_columns: - home_zone_id - income @@ -68,6 +70,9 @@ input_table_list: rename_columns: MAZ: zone_id COUNTY: county_id + recode_columns: + zone_id: zero-based + TAZ: land_use_taz.TAZ keep_columns: - TAZ - DISTRICT @@ -95,6 +100,11 @@ input_table_list: - TOPOLOGY - TERMINAL # - access_dist_transit + - tablename: land_use_taz + filename: taz.csv + index_col: TAZ + recode_columns: + TAZ: zero-based models: - initialize_landuse @@ -140,11 +150,25 @@ output_tables: tables: - checkpoints - accessibility - - land_use - - households - - persons - - tours - - trips + - tablename: land_use + decode_columns: + zone_id: land_use.zone_id + - tablename: households + decode_columns: + home_zone_id: land_use.zone_id + - tablename: persons + decode_columns: + home_zone_id: land_use.zone_id + school_zone_id: nonnegative | land_use.zone_id + workplace_zone_id: nonnegative | land_use.zone_id + - tablename: tours + decode_columns: + origin: land_use.zone_id + destination: land_use.zone_id + - tablename: trips + decode_columns: + origin: land_use.zone_id + destination: land_use.zone_id - joint_tour_participants # area_types less than this are considered urban diff --git a/activitysim/examples/placeholder_sandag/configs_3_zone/legacy-1.0.4/logging.yaml b/activitysim/examples/placeholder_sandag/configs_3_zone/legacy-1.0.4/logging.yaml new file mode 100644 index 0000000000..3d62e8064b --- /dev/null +++ b/activitysim/examples/placeholder_sandag/configs_3_zone/legacy-1.0.4/logging.yaml @@ -0,0 +1,54 @@ +# Config for logging +# ------------------ +# See http://docs.python.org/2.7/library/logging.config.html#configuration-dictionary-schema + +logging: + version: 1 + disable_existing_loggers: true + + + # Configuring the default (root) logger is highly recommended + root: + level: NOTSET + handlers: [console, logfile] + + loggers: + + activitysim: + level: INFO + handlers: [console, logfile] + propagate: false + + orca: + level: WARNING + handlers: [console, logfile] + propagate: false + + handlers: + + logfile: + class: logging.FileHandler + filename: !!python/object/apply:activitysim.core.config.log_file_path ['activitysim.log'] + mode: w + formatter: fileFormatter + level: NOTSET + + console: + class: logging.StreamHandler + stream: ext://sys.stdout + formatter: simpleFormatter + level: !!python/object/apply:activitysim.core.mp_tasks.if_sub_task [WARNING, NOTSET] + + formatters: + + simpleFormatter: + class: logging.Formatter + format: !!python/object/apply:activitysim.core.mp_tasks.if_sub_task [ + '%(processName)-10s %(levelname)s - %(name)s - %(message)s', + '%(levelname)s - %(name)s - %(message)s'] + datefmt: '%d/%m/%Y %H:%M:%S' + + fileFormatter: + class: logging.Formatter + format: '%(asctime)s - %(levelname)s - %(name)s - %(message)s' + datefmt: '%d/%m/%Y %H:%M:%S' diff --git a/activitysim/examples/placeholder_sandag/configs_3_zone/legacy-1.0.4/network_los.yaml b/activitysim/examples/placeholder_sandag/configs_3_zone/legacy-1.0.4/network_los.yaml new file mode 100644 index 0000000000..64bd95fd07 --- /dev/null +++ b/activitysim/examples/placeholder_sandag/configs_3_zone/legacy-1.0.4/network_los.yaml @@ -0,0 +1,164 @@ +inherit_settings: True + +zone_system: 3 + +skim_dict_factory: NumpyArraySkimFactory +#skim_dict_factory: MemMapSkimFactory + +# read cached skims (using numpy memmap) from output directory (memmap is faster than omx ) +read_skim_cache: False +# write memmapped cached skims to output directory after reading from omx, for use in subsequent runs +write_skim_cache: False + +# rebuild and overwrite existing tap_tap_utilities cache +rebuild_tvpb_cache: True + +# write a csv version of tvpb cache for tracing when checkpointing cache. +# (writes csv file when writing/checkpointing cache i.e. when cached changed) +# (n.b. csv file could be quite large if cache is STATIC!) +trace_tvpb_cache_as_csv: False + +taz_skims: taz_skims*.omx + +# we require that skims for all tap_tap sets have unique names +# and can therefor share a single skim_dict without name collision +# e.g. TRN_XWAIT_FAST__AM, TRN_XWAIT_SHORT__AM, TRN_XWAIT_CHEAP__AM +tap_skims: tap_skims*.omx + +maz: maz.csv + +tap: tap.csv + +tap_lines: tap_lines.csv + +maz_to_maz: + tables: + - maz_to_maz_walk.csv + - maz_to_maz_bike.csv + + # maz_to_maz blending distance (missing or 0 means no blending) + max_blend_distance: + # blend distance of 0 means no blending + DISTBIKE: 0 + DISTWALK: 1 + + # missing means use the skim value itself rather than DIST skim (e.g. DISTBIKE) + blend_distance_skim_name: DIST + +maz_to_tap: + walk: + table: maz_to_tap_walk.csv + # if provided, this column will be used (together with tap_lines table) to trim the near tap set + # to only include the nearest tap to origin when more than oFne tap serves the same line + tap_line_distance_col: DISTWALK + max_dist: 1.2 + drive: + table: maz_to_tap_drive.csv + # although maz_tap untility calculations use both DIST and drive_time, + # it looks like drive_time is a linera function of DIST so we can use either column to trim to nearest + tap_line_distance_col: DIST + +skim_time_periods: + time_window: 1440 + period_minutes: 60 + periods: [0, 6, 11, 16, 20, 24] + labels: &skim_time_period_labels ['EA', 'AM', 'MD', 'PM', 'EV'] + +demographic_segments: &demographic_segments + - &low_income_segment_id 0 + - &high_income_segment_id 1 + +# transit virtual path builder settings +TVPB_SETTINGS: + + tour_mode_choice: + units: utility + path_types: + WTW: + access: walk + egress: walk + max_paths_across_tap_sets: 3 + max_paths_per_tap_set: 1 + DTW: + access: drive + egress: walk + max_paths_across_tap_sets: 3 + max_paths_per_tap_set: 1 + WTD: + access: walk + egress: drive + max_paths_across_tap_sets: 3 + max_paths_per_tap_set: 1 + tap_tap_settings: + SPEC: tvpb_utility_tap_tap.csv + PREPROCESSOR: + SPEC: tvpb_utility_tap_tap_annotate_choosers_preprocessor.csv + DF: df + # FIXME this has to be explicitly specified, since e.g. attribute columns are assigned in expression files + attribute_segments: + demographic_segment: *demographic_segments + tod: *skim_time_period_labels + access_mode: ['drive', 'walk'] + attributes_as_columns: + - demographic_segment + - tod + maz_tap_settings: + walk: + SPEC: tvpb_utility_walk_maz_tap.csv + CHOOSER_COLUMNS: + #- demographic_segment + - walk_time + drive: + SPEC: tvpb_utility_drive_maz_tap.csv + CHOOSER_COLUMNS: + #- demographic_segment + - drive_time + - DIST + CONSTANTS: + C_LOW_INCOME_SEGMENT_ID: *low_income_segment_id + C_HIGH_INCOME_SEGMENT_ID: *high_income_segment_id + TVPB_demographic_segments_by_income_segment: + 1: *low_income_segment_id + 2: *low_income_segment_id + 3: *high_income_segment_id + 4: *high_income_segment_id + c_ivt_high_income: -0.028 + c_ivt_low_income: -0.0175 + c_cost_high_income: -0.00112 + c_cost_low_income: -0.00112 + c_wait: 1.5 + c_walk: 1.7 + c_drive: 1.5 + c_auto_operating_cost_per_mile: 18.29 + C_UNAVAILABLE: -999 + C_FASTEST_IVT_MULTIPLIER: 2 + C_FASTEST_COST_MULTIPLIER: 1 + C_CHEAPEST_IVT_MULTIPLIER: 1 + C_CHEAPEST_COST_MULTIPLIER: 500 + C_SHORTEST_IVT_MULTIPLIER: 1 + C_SHORTEST_COST_MULTIPLIER: 1 + C_SHORTEST_DIST_MULTIPLIER: 1 + # illustrate using access mode in tat-tap expressions files + C_DRIVE_TRANSFER_PENALTY: -1 + + accessibility: + units: time + path_types: + WTW: + access: walk + egress: walk + max_paths_across_tap_sets: 1 + max_paths_per_tap_set: 1 + tap_tap_settings: + SPEC: tvpb_accessibility_tap_tap_.csv + # FIXME this has to be explicitly specified, since e.g. attribute columns are assigned in expression files + attribute_segments: + tod: *skim_time_period_labels + maz_tap_settings: + walk: + SPEC: tvpb_accessibility_walk_maz_tap.csv + CHOOSER_COLUMNS: + - walk_time + CONSTANTS: + out_of_vehicle_walk_time_weight: 1.5 + out_of_vehicle_wait_time_weight: 2.0 diff --git a/activitysim/examples/placeholder_sandag/configs_3_zone/legacy-1.0.4/settings.yaml b/activitysim/examples/placeholder_sandag/configs_3_zone/legacy-1.0.4/settings.yaml new file mode 100644 index 0000000000..d569385c09 --- /dev/null +++ b/activitysim/examples/placeholder_sandag/configs_3_zone/legacy-1.0.4/settings.yaml @@ -0,0 +1,131 @@ +inherit_settings: True + +# - tracing + +# trace household id; comment out or leave empty for no trace +# households with all tour types +trace_hh_id: + +# trace origin, destination in accessibility calculation; comment out or leave empty for no trace +trace_od: + +# input tables +input_table_list: + - tablename: households + filename: households.csv + index_col: household_id + rename_columns: + HHID: household_id + PERSONS: hhsize + workers: num_workers + VEHICL: auto_ownership + MAZ: home_zone_id + keep_columns: + - home_zone_id + - income + - hhsize + - HHT + - auto_ownership + - num_workers + - tablename: persons + filename: persons.csv + index_col: person_id + rename_columns: + PERID: person_id + keep_columns: + - household_id + - age + - PNUM + - sex + - pemploy + - pstudent + - ptype + - tablename: land_use + filename: land_use.csv + index_col: zone_id + rename_columns: + MAZ: zone_id + COUNTY: county_id + keep_columns: + - TAZ + - DISTRICT + - SD + - county_id + - TOTHH + - TOTPOP + - TOTACRE + - RESACRE + - CIACRE + - TOTEMP + - AGE0519 + - RETEMPN + - FPSEMPN + - HEREMPN + - OTHEMPN + - AGREMPN + - MWTEMPN + - PRKCST + - OPRKCST + - area_type + - HSENROLL + - COLLFTE + - COLLPTE + - TOPOLOGY + - TERMINAL + - access_dist_transit + +output_tables: + h5_store: False + action: include + prefix: final_ + sort: True + tables: + - checkpoints + - accessibility + - land_use + - households + - persons + - tours + - trips + +models: + - initialize_landuse + - initialize_households + - compute_accessibility + # --- STATIC cache prebuild steps + # single-process step to create attribute_combination list + - initialize_los + # multi-processable step to build STATIC cache + # (this step is a NOP if cache already exists and network_los.rebuild_tvpb_cache setting is False) + - initialize_tvpb + # --- + - school_location + - workplace_location + - auto_ownership_simulate + - free_parking + - cdap_simulate + - mandatory_tour_frequency + - mandatory_tour_scheduling + - joint_tour_frequency + - joint_tour_composition + - joint_tour_participation + - joint_tour_destination + - joint_tour_scheduling + - non_mandatory_tour_frequency + - non_mandatory_tour_destination + - non_mandatory_tour_scheduling + - tour_mode_choice_simulate + - atwork_subtour_frequency + - atwork_subtour_destination + - atwork_subtour_scheduling + - atwork_subtour_mode_choice + - stop_frequency + - trip_purpose + - trip_destination + - trip_purpose_and_destination + - trip_scheduling + - trip_mode_choice + - write_data_dictionary + - track_skim_usage + - write_trip_matrices + - write_tables diff --git a/activitysim/examples/placeholder_sandag/configs_3_zone/legacy-1.0.4/settings_mp.yaml b/activitysim/examples/placeholder_sandag/configs_3_zone/legacy-1.0.4/settings_mp.yaml new file mode 100644 index 0000000000..3642cb1e15 --- /dev/null +++ b/activitysim/examples/placeholder_sandag/configs_3_zone/legacy-1.0.4/settings_mp.yaml @@ -0,0 +1,34 @@ +inherit_settings: settings.yaml + +multiprocess: True +households_sample_size: 0 +num_processes: 28 + +# raise error if any sub-process fails without waiting for others to complete +# (Shadow pricing requires fail_fast setting in multiprocessing mode) +fail_fast: True + +multiprocess_steps: + - name: mp_initialize + begin: initialize_landuse + - name: mp_accessibility + begin: compute_accessibility + slice: + tables: + - accessibility + - name: mp_los + begin: initialize_los + - name: mp_tvpb + begin: initialize_tvpb + num_processes: 20 + slice: + tables: + - attribute_combinations + - name: mp_models + begin: school_location + slice: + tables: + - households + - persons + - name: mp_summarize + begin: write_data_dictionary diff --git a/activitysim/examples/placeholder_sandag/configs_3_zone/legacy-1.0.4/trip_destination_annotate_trips_preprocessor.csv b/activitysim/examples/placeholder_sandag/configs_3_zone/legacy-1.0.4/trip_destination_annotate_trips_preprocessor.csv new file mode 100644 index 0000000000..9db6a951da --- /dev/null +++ b/activitysim/examples/placeholder_sandag/configs_3_zone/legacy-1.0.4/trip_destination_annotate_trips_preprocessor.csv @@ -0,0 +1,12 @@ +Description,Target,Expression +#,, +,tour_mode,"reindex(tours.tour_mode, df.tour_id)" +,_tod,"np.where(df.outbound,reindex_i(tours.start, df.tour_id),reindex_i(tours.end, df.tour_id))" +,trip_period,network_los.skim_time_period_label(_tod) +,is_joint,"reindex(tours.tour_category, df.tour_id)=='joint'" +#,,not needed as school is not chosen as an intermediate trip destination +#,_grade_school,"(df.primary_purpose == 'school') & reindex(persons.is_gradeschool, df.person_id)" +#,size_segment,"df.primary_purpose.where(df.primary_purpose != 'school', np.where(_grade_school,'gradeschool', 'highschool'))" +,tour_mode_is_walk,"reindex(tours.tour_mode, df.tour_id)=='WALK'" +,tour_mode_is_bike,"reindex(tours.tour_mode, df.tour_id)=='BIKE'" +,tour_leg_dest,"np.where(df.outbound,reindex(tours.destination, df.tour_id), reindex(tours.origin, df.tour_id)).astype(int)" diff --git a/activitysim/examples/placeholder_sandag/configs_3_zone/settings.yaml b/activitysim/examples/placeholder_sandag/configs_3_zone/settings.yaml index ad373d8ff7..97221aca7a 100644 --- a/activitysim/examples/placeholder_sandag/configs_3_zone/settings.yaml +++ b/activitysim/examples/placeholder_sandag/configs_3_zone/settings.yaml @@ -9,6 +9,9 @@ trace_hh_id: # trace origin, destination in accessibility calculation; comment out or leave empty for no trace trace_od: +# presampling +want_dest_choice_presampling: True + # input tables input_table_list: - tablename: households @@ -20,6 +23,8 @@ input_table_list: workers: num_workers VEHICL: auto_ownership MAZ: home_zone_id + recode_columns: + home_zone_id: land_use.zone_id keep_columns: - home_zone_id - income @@ -46,6 +51,9 @@ input_table_list: rename_columns: MAZ: zone_id COUNTY: county_id + recode_columns: + zone_id: zero-based + TAZ: land_use_taz.TAZ keep_columns: - TAZ - DISTRICT @@ -73,6 +81,11 @@ input_table_list: - TOPOLOGY - TERMINAL - access_dist_transit + - tablename: land_use_taz + filename: taz.csv + index_col: TAZ + recode_columns: + TAZ: zero-based output_tables: h5_store: False @@ -82,11 +95,26 @@ output_tables: tables: - checkpoints - accessibility - - land_use - - households - - persons - - tours - - trips + - tablename: land_use + decode_columns: + zone_id: land_use.zone_id + - tablename: households + decode_columns: + home_zone_id: land_use.zone_id + - tablename: persons + decode_columns: + home_zone_id: land_use.zone_id + school_zone_id: nonnegative | land_use.zone_id + workplace_zone_id: nonnegative | land_use.zone_id + - tablename: tours + decode_columns: + origin: land_use.zone_id + destination: land_use.zone_id + - tablename: trips + decode_columns: + origin: land_use.zone_id + destination: land_use.zone_id + - joint_tour_participants models: - initialize_landuse @@ -129,7 +157,3 @@ models: - track_skim_usage - write_trip_matrices - write_tables - - - - diff --git a/activitysim/examples/placeholder_sandag/configs_3_zone/trip_mode_choice.yaml b/activitysim/examples/placeholder_sandag/configs_3_zone/trip_mode_choice.yaml index 081d5d8429..cdb5c7fef9 100644 --- a/activitysim/examples/placeholder_sandag/configs_3_zone/trip_mode_choice.yaml +++ b/activitysim/examples/placeholder_sandag/configs_3_zone/trip_mode_choice.yaml @@ -76,16 +76,8 @@ CONSTANTS: SHARED3PAY: 6 WALK: 7 BIKE: 8 - WALK_LOC: 9 - WALK_LRF: 10 - WALK_EXP: 11 - WALK_HVY: 12 - WALK_COM: 13 - DRIVE_LOC: 14 - DRIVE_LRF: 15 - DRIVE_EXP: 16 - DRIVE_HVY: 17 - DRIVE_COM: 18 + WALK_TRANSIT: 9 + DRIVE_TRANSIT: 14 TAXI: 19 TNC_SINGLE: 20 TNC_SHARED: 21 @@ -95,8 +87,8 @@ CONSTANTS: I_AUTO_MODES: [1, 2, 3, 4, 5, 6] I_WALK_MODE: 7 I_BIKE_MODE: 8 - I_WALK_TRANSIT_MODES: [9, 10, 11, 12, 13] - I_DRIVE_TRANSIT_MODES: [14, 15, 16, 17, 18] + I_WALK_TRANSIT_MODES: [9] + I_DRIVE_TRANSIT_MODES: [14] I_RIDE_HAIL_MODES: [19, 20, 21] # RIDEHAIL Settings Taxi_baseFare: 2.20 @@ -108,7 +100,7 @@ CONSTANTS: 3: 13.3 4: 17.3 5: 26.5 - Taxi_waitTime_sd: + Taxi_waitTime_sd: 1: 0 2: 0 3: 0 @@ -118,13 +110,13 @@ CONSTANTS: TNC_single_costPerMile: 1.33 TNC_single_costPerMinute: 0.24 TNC_single_costMinimum: 7.20 - TNC_single_waitTime_mean: + TNC_single_waitTime_mean: 1: 3.0 2: 6.3 3: 8.4 4: 8.5 5: 10.3 - TNC_single_waitTime_sd: + TNC_single_waitTime_sd: 1: 0 2: 0 3: 0 @@ -135,13 +127,13 @@ CONSTANTS: TNC_shared_costPerMinute: 0.10 TNC_shared_costMinimum: 3.00 TNC_shared_IVTFactor: 1.5 - TNC_shared_waitTime_mean: + TNC_shared_waitTime_mean: 1: 5.0 2: 8.0 3: 11.0 4: 15.0 5: 15.0 - TNC_shared_waitTime_sd: + TNC_shared_waitTime_sd: 1: 0 2: 0 3: 0 diff --git a/activitysim/examples/placeholder_sandag/test/configs_1_sharrow/network_los.yaml b/activitysim/examples/placeholder_sandag/test/configs_1_sharrow/network_los.yaml new file mode 100644 index 0000000000..1c4cd79daf --- /dev/null +++ b/activitysim/examples/placeholder_sandag/test/configs_1_sharrow/network_los.yaml @@ -0,0 +1,6 @@ +inherit_settings: True + +# read cached skims (using numpy memmap) from output directory (memmap is faster than omx ) +read_skim_cache: False +# write memmapped cached skims to output directory after reading from omx, for use in subsequent runs +write_skim_cache: False diff --git a/activitysim/examples/placeholder_sandag/test/configs_1_sharrow/settings.yaml b/activitysim/examples/placeholder_sandag/test/configs_1_sharrow/settings.yaml new file mode 100644 index 0000000000..5ad20bda76 --- /dev/null +++ b/activitysim/examples/placeholder_sandag/test/configs_1_sharrow/settings.yaml @@ -0,0 +1,19 @@ +inherit_settings: True + +output_tables: + h5_store: False + action: include + prefix: final_1_zone_ + sort: True + tables: + - tablename: trips + decode_columns: + origin: land_use.zone_id + destination: land_use.zone_id + - tablename: tours + decode_columns: + origin: land_use.zone_id + destination: land_use.zone_id + +sharrow: require +recode_pipeline_columns: True diff --git a/activitysim/examples/placeholder_sandag/test/configs_1_zone/settings.yaml b/activitysim/examples/placeholder_sandag/test/configs_1_zone/settings.yaml index bf0e418e96..92b4d430a0 100644 --- a/activitysim/examples/placeholder_sandag/test/configs_1_zone/settings.yaml +++ b/activitysim/examples/placeholder_sandag/test/configs_1_zone/settings.yaml @@ -24,3 +24,5 @@ output_tables: tables: - trips - tours + +recode_pipeline_columns: False diff --git a/activitysim/examples/placeholder_sandag/test/configs_2_sharrow/network_los.yaml b/activitysim/examples/placeholder_sandag/test/configs_2_sharrow/network_los.yaml new file mode 100644 index 0000000000..1c4cd79daf --- /dev/null +++ b/activitysim/examples/placeholder_sandag/test/configs_2_sharrow/network_los.yaml @@ -0,0 +1,6 @@ +inherit_settings: True + +# read cached skims (using numpy memmap) from output directory (memmap is faster than omx ) +read_skim_cache: False +# write memmapped cached skims to output directory after reading from omx, for use in subsequent runs +write_skim_cache: False diff --git a/activitysim/examples/placeholder_sandag/test/configs_2_sharrow/settings.yaml b/activitysim/examples/placeholder_sandag/test/configs_2_sharrow/settings.yaml new file mode 100644 index 0000000000..446b36eeff --- /dev/null +++ b/activitysim/examples/placeholder_sandag/test/configs_2_sharrow/settings.yaml @@ -0,0 +1,19 @@ +inherit_settings: True + +output_tables: + h5_store: False + action: include + prefix: final_2_zone_ + sort: True + tables: + - tablename: trips + decode_columns: + origin: land_use.zone_id + destination: land_use.zone_id + - tablename: tours + decode_columns: + origin: land_use.zone_id + destination: land_use.zone_id + +sharrow: require +recode_pipeline_columns: True diff --git a/activitysim/examples/placeholder_sandag/test/configs_2_zone/settings.yaml b/activitysim/examples/placeholder_sandag/test/configs_2_zone/settings.yaml index 5625718faf..cb6166c47d 100644 --- a/activitysim/examples/placeholder_sandag/test/configs_2_zone/settings.yaml +++ b/activitysim/examples/placeholder_sandag/test/configs_2_zone/settings.yaml @@ -27,3 +27,5 @@ output_tables: tables: - trips - tours + +recode_pipeline_columns: False diff --git a/activitysim/examples/placeholder_sandag/test/configs_3_sharrow/network_los.yaml b/activitysim/examples/placeholder_sandag/test/configs_3_sharrow/network_los.yaml new file mode 100644 index 0000000000..1c4cd79daf --- /dev/null +++ b/activitysim/examples/placeholder_sandag/test/configs_3_sharrow/network_los.yaml @@ -0,0 +1,6 @@ +inherit_settings: True + +# read cached skims (using numpy memmap) from output directory (memmap is faster than omx ) +read_skim_cache: False +# write memmapped cached skims to output directory after reading from omx, for use in subsequent runs +write_skim_cache: False diff --git a/activitysim/examples/placeholder_sandag/test/configs_3_sharrow/settings.yaml b/activitysim/examples/placeholder_sandag/test/configs_3_sharrow/settings.yaml new file mode 100644 index 0000000000..57445acbcb --- /dev/null +++ b/activitysim/examples/placeholder_sandag/test/configs_3_sharrow/settings.yaml @@ -0,0 +1,19 @@ +inherit_settings: True + +output_tables: + h5_store: False + action: include + prefix: final_3_zone_ + sort: True + tables: + - tablename: trips + decode_columns: + origin: land_use.zone_id + destination: land_use.zone_id + - tablename: tours + decode_columns: + origin: land_use.zone_id + destination: land_use.zone_id + +sharrow: test +recode_pipeline_columns: True diff --git a/activitysim/examples/placeholder_sandag/test/output_1/.gitignore b/activitysim/examples/placeholder_sandag/test/output_1/.gitignore new file mode 100644 index 0000000000..bf5bf15e3e --- /dev/null +++ b/activitysim/examples/placeholder_sandag/test/output_1/.gitignore @@ -0,0 +1,7 @@ +*.csv +*.log +*.prof +*.h5 +*.txt +*.yaml +*.omx diff --git a/activitysim/examples/placeholder_sandag/test/output_1/cache/.gitignore b/activitysim/examples/placeholder_sandag/test/output_1/cache/.gitignore new file mode 100644 index 0000000000..3dd2e62f9e --- /dev/null +++ b/activitysim/examples/placeholder_sandag/test/output_1/cache/.gitignore @@ -0,0 +1,2 @@ +*.mmap +*.feather diff --git a/activitysim/examples/placeholder_sandag/test/output_1/trace/.gitignore b/activitysim/examples/placeholder_sandag/test/output_1/trace/.gitignore new file mode 100644 index 0000000000..8edb806780 --- /dev/null +++ b/activitysim/examples/placeholder_sandag/test/output_1/trace/.gitignore @@ -0,0 +1,3 @@ +*.csv +*.log +*.txt diff --git a/activitysim/examples/placeholder_sandag/test/output_2/.gitignore b/activitysim/examples/placeholder_sandag/test/output_2/.gitignore new file mode 100644 index 0000000000..bf5bf15e3e --- /dev/null +++ b/activitysim/examples/placeholder_sandag/test/output_2/.gitignore @@ -0,0 +1,7 @@ +*.csv +*.log +*.prof +*.h5 +*.txt +*.yaml +*.omx diff --git a/activitysim/examples/placeholder_sandag/test/output_2/cache/.gitignore b/activitysim/examples/placeholder_sandag/test/output_2/cache/.gitignore new file mode 100644 index 0000000000..3dd2e62f9e --- /dev/null +++ b/activitysim/examples/placeholder_sandag/test/output_2/cache/.gitignore @@ -0,0 +1,2 @@ +*.mmap +*.feather diff --git a/activitysim/examples/placeholder_sandag/test/output_2/trace/.gitignore b/activitysim/examples/placeholder_sandag/test/output_2/trace/.gitignore new file mode 100644 index 0000000000..8edb806780 --- /dev/null +++ b/activitysim/examples/placeholder_sandag/test/output_2/trace/.gitignore @@ -0,0 +1,3 @@ +*.csv +*.log +*.txt diff --git a/activitysim/examples/placeholder_sandag/test/output_3/.gitignore b/activitysim/examples/placeholder_sandag/test/output_3/.gitignore new file mode 100644 index 0000000000..bf5bf15e3e --- /dev/null +++ b/activitysim/examples/placeholder_sandag/test/output_3/.gitignore @@ -0,0 +1,7 @@ +*.csv +*.log +*.prof +*.h5 +*.txt +*.yaml +*.omx diff --git a/activitysim/examples/placeholder_sandag/test/output_3/cache/.gitignore b/activitysim/examples/placeholder_sandag/test/output_3/cache/.gitignore new file mode 100644 index 0000000000..3dd2e62f9e --- /dev/null +++ b/activitysim/examples/placeholder_sandag/test/output_3/cache/.gitignore @@ -0,0 +1,2 @@ +*.mmap +*.feather diff --git a/activitysim/examples/placeholder_sandag/test/output_3/trace/.gitignore b/activitysim/examples/placeholder_sandag/test/output_3/trace/.gitignore new file mode 100644 index 0000000000..8edb806780 --- /dev/null +++ b/activitysim/examples/placeholder_sandag/test/output_3/trace/.gitignore @@ -0,0 +1,3 @@ +*.csv +*.log +*.txt diff --git a/activitysim/examples/placeholder_sandag/test/regress/.gitignore b/activitysim/examples/placeholder_sandag/test/regress/.gitignore new file mode 100644 index 0000000000..c5200d0c42 --- /dev/null +++ b/activitysim/examples/placeholder_sandag/test/regress/.gitignore @@ -0,0 +1 @@ +*_last_run.csv diff --git a/activitysim/examples/placeholder_sandag/test/regress/final_1_zone_tours.csv b/activitysim/examples/placeholder_sandag/test/regress/final_1_zone_tours.csv index 84449693e1..c5f84186bb 100644 --- a/activitysim/examples/placeholder_sandag/test/regress/final_1_zone_tours.csv +++ b/activitysim/examples/placeholder_sandag/test/regress/final_1_zone_tours.csv @@ -1,63 +1,63 @@ tour_id,person_id,tour_type,tour_type_count,tour_type_num,tour_num,tour_count,tour_category,number_of_participants,destination,origin,household_id,tdd,start,end,duration,composition,destination_logsum,tour_mode,mode_choice_logsum,atwork_subtour_frequency,parent_tour_id,stop_frequency,primary_purpose -1359025,33146,work,1,1,1,1,mandatory,1,3131.0,3259.0,12593,11.0,5.0,16.0,11.0,,,SHARED2FREE,0.0449709966797433,no_subtours,,2out_0in,work +1359025,33146,work,1,1,1,1,mandatory,1,3131.0,3259.0,12593,11.0,5.0,16.0,11.0,,,SHARED2FREE,0.04497099667974337,no_subtours,,2out_0in,work 1359066,33147,work,1,1,1,1,mandatory,1,3342.0,3259.0,12593,59.0,8.0,13.0,5.0,,,SHARED2FREE,0.923483008481964,no_subtours,,0out_3in,work -1494647,36454,shopping,2,1,1,2,non_mandatory,1,3342.0,3134.0,13797,135.0,14.0,14.0,0.0,,11.401050563880888,DRIVEALONEFREE,0.2595703435486286,,,0out_1in,shopping -1494648,36454,shopping,2,2,2,2,non_mandatory,1,3497.0,3134.0,13797,154.0,16.0,16.0,0.0,,11.439449857137804,SHARED2FREE,-0.0006093410329746,,,1out_0in,shopping +1494647,36454,shopping,2,1,1,2,non_mandatory,1,3342.0,3134.0,13797,135.0,14.0,14.0,0.0,,11.401050563880887,DRIVEALONEFREE,0.2595703435486286,,,0out_1in,shopping +1494648,36454,shopping,2,2,2,2,non_mandatory,1,3497.0,3134.0,13797,154.0,16.0,16.0,0.0,,11.439449857137804,SHARED2FREE,-0.0006093410329746202,,,1out_0in,shopping 1494694,36455,work,1,1,1,1,mandatory,1,3317.0,3134.0,13797,53.0,7.0,23.0,16.0,,,SHARED3FREE,0.2738771147247297,no_subtours,,0out_0in,work -1709911,41705,eatout,1,1,1,1,non_mandatory,1,3437.0,3342.0,15777,85.0,10.0,10.0,0.0,,11.909980318826896,SHARED2FREE,0.7916769643872005,,,0out_0in,eatout +1709911,41705,eatout,1,1,1,1,non_mandatory,1,3437.0,3342.0,15777,85.0,10.0,10.0,0.0,,11.909980318826895,SHARED2FREE,0.7916769643872005,,,0out_0in,eatout 1709946,41706,business,1,1,1,1,atwork,1,3131.0,3398.0,15777,117.0,12.0,17.0,5.0,,18.502192217204307,SHARED2FREE,-1.0113258602519364,,1709985.0,0out_0in,atwork 1709985,41706,work,1,1,1,1,mandatory,1,3398.0,3342.0,15777,65.0,8.0,19.0,11.0,,,DRIVEALONEFREE,0.5212273251520757,business1,,0out_1in,work -2051456,50035,social,1,1,1,1,joint,2,3497.0,3493.0,18261,146.0,15.0,16.0,1.0,mixed,11.601381369295757,SHARED2FREE,-1.4037659073121551,,,0out_2in,social +2051456,50035,social,1,1,1,1,joint,2,3497.0,3493.0,18261,146.0,15.0,16.0,1.0,mixed,11.601381369295755,SHARED2FREE,-1.4037659073121553,,,0out_2in,social 2051466,50035,school,1,1,1,1,mandatory,1,3340.0,3493.0,18261,42.0,7.0,12.0,5.0,,,SHARED3FREE,-1.3497203179845243,,,0out_0in,school -2051468,50035,shopping,1,1,1,1,non_mandatory,1,3396.0,3493.0,18261,172.0,18.0,21.0,3.0,,11.366590261069064,DRIVEALONEFREE,0.6130171387405682,,,0out_0in,shopping -2051509,50036,shopping,1,1,1,1,non_mandatory,1,3497.0,3493.0,18261,99.0,11.0,11.0,0.0,,11.55566952538272,DRIVEALONEFREE,0.6647948166112536,,,0out_1in,shopping +2051468,50035,shopping,1,1,1,1,non_mandatory,1,3396.0,3493.0,18261,172.0,18.0,21.0,3.0,,11.366590261069065,DRIVEALONEFREE,0.6130171387405682,,,0out_0in,shopping +2051509,50036,shopping,1,1,1,1,non_mandatory,1,3497.0,3493.0,18261,99.0,11.0,11.0,0.0,,11.555669525382719,DRIVEALONEFREE,0.6647948166112536,,,0out_1in,shopping 2051556,50037,work,1,1,1,1,mandatory,1,3396.0,3493.0,18261,44.0,7.0,14.0,7.0,,,DRIVEALONEFREE,0.5299854110237027,no_subtours,,0out_0in,work -2268889,55338,school,1,1,1,1,mandatory,1,3592.0,3592.0,19758,40.0,7.0,10.0,3.0,,,WALK,0.3318320919975738,,,0out_0in,school +2268889,55338,school,1,1,1,1,mandatory,1,3592.0,3592.0,19758,40.0,7.0,10.0,3.0,,,WALK,0.33183209199757385,,,0out_0in,school 2268938,55339,work,1,1,1,1,mandatory,1,3497.0,3592.0,19758,11.0,5.0,16.0,11.0,,,WALK,0.537772515990635,no_subtours,,0out_0in,work 2373816,57897,work,1,1,1,1,mandatory,1,3176.0,3746.0,20552,50.0,7.0,20.0,13.0,,,DRIVEALONEFREE,-0.28388579338221,no_subtours,,0out_0in,work 2373822,57898,eat,1,1,1,1,atwork,1,3410.0,3347.0,20552,125.0,13.0,14.0,1.0,,11.100624000564997,DRIVEALONEFREE,0.7681389958489114,,2373857.0,0out_0in,atwork 2373857,57898,work,1,1,1,1,mandatory,1,3347.0,3746.0,20552,103.0,11.0,15.0,4.0,,,DRIVEALONEFREE,0.520379765074684,eat,,0out_1in,work 2373898,57899,work,1,1,1,1,mandatory,1,3402.0,3746.0,20552,47.0,7.0,17.0,10.0,,,WALK,1.0388895039783694,no_subtours,,0out_0in,work -2373980,57901,work,2,1,1,2,mandatory,1,3115.0,3746.0,20552,24.0,6.0,11.0,5.0,,,SHARED3FREE,0.6022315390131013,no_subtours,,0out_0in,work -2373981,57901,work,2,2,2,2,mandatory,1,3115.0,3746.0,20552,148.0,15.0,18.0,3.0,,,SHARED2FREE,0.6101482774173842,no_subtours,,1out_0in,work +2373980,57901,work,2,1,1,2,mandatory,1,3115.0,3746.0,20552,25.0,6.0,12.0,6.0,,,SHARED3FREE,0.6022315390131013,no_subtours,,0out_0in,work +2373981,57901,work,2,2,2,2,mandatory,1,3115.0,3746.0,20552,150.0,15.0,20.0,5.0,,,SHARED2FREE,0.6232767878249469,no_subtours,,1out_0in,work 2563802,62531,school,1,1,1,1,mandatory,1,3460.0,3316.0,21869,180.0,20.0,20.0,0.0,,,SHARED3FREE,-0.7094603590463964,,,0out_0in,school -2563821,62532,escort,1,1,1,1,non_mandatory,1,3398.0,3316.0,21869,20.0,6.0,7.0,1.0,,12.499268454965652,SHARED2FREE,-1.46041546280727,,,0out_0in,escort +2563821,62532,escort,1,1,1,1,non_mandatory,1,3398.0,3316.0,21869,20.0,6.0,7.0,1.0,,12.499268454965652,SHARED2FREE,-1.4604154628072699,,,0out_0in,escort 2563862,62533,escort,3,1,1,4,non_mandatory,1,3402.0,3316.0,21869,1.0,5.0,6.0,1.0,,12.534424209198946,SHARED3FREE,-1.2940574569954848,,,0out_3in,escort -2563863,62533,escort,3,2,2,4,non_mandatory,1,3519.0,3316.0,21869,99.0,11.0,11.0,0.0,,12.466623656700463,SHARED2FREE,-0.9326373013150776,,,0out_0in,escort +2563863,62533,escort,3,2,2,4,non_mandatory,1,3519.0,3316.0,21869,99.0,11.0,11.0,0.0,,12.466623656700463,SHARED2FREE,-0.9326373013150777,,,0out_0in,escort 2563864,62533,escort,3,3,3,4,non_mandatory,1,3398.0,3316.0,21869,135.0,14.0,14.0,0.0,,12.485628371830364,SHARED3FREE,-1.4467093822325976,,,0out_0in,escort -2563878,62533,othdiscr,1,1,4,4,non_mandatory,1,3383.0,3316.0,21869,99.0,11.0,11.0,0.0,,12.893824062601642,DRIVEALONEFREE,0.0900147292900378,,,0out_0in,othdiscr +2563878,62533,othdiscr,1,1,4,4,non_mandatory,1,3383.0,3316.0,21869,99.0,11.0,11.0,0.0,,12.893824062601642,DRIVEALONEFREE,0.09001472929003788,,,0out_0in,othdiscr 2563925,62534,school,1,1,1,1,mandatory,1,3316.0,3316.0,21869,55.0,8.0,9.0,1.0,,,SHARED3FREE,0.4282867689528016,,,0out_1in,school -2787968,67999,escort,1,1,1,2,non_mandatory,1,3410.0,3378.0,23619,124.0,13.0,13.0,0.0,,12.955709049606323,TNC_SINGLE,-0.5513480052414359,,,0out_2in,escort +2787968,67999,escort,1,1,1,2,non_mandatory,1,3410.0,3378.0,23619,124.0,13.0,13.0,0.0,,12.955709049606325,TNC_SINGLE,-0.5513480052414359,,,0out_2in,escort 2787995,67999,social,1,1,2,2,non_mandatory,1,3400.0,3378.0,23619,165.0,17.0,20.0,3.0,,11.998752822352907,WALK,0.8829112299755879,,,0out_0in,social 2788039,68000,work,1,1,1,1,mandatory,1,3375.0,3378.0,23619,50.0,7.0,20.0,13.0,,,WALK,0.6308240193619252,no_subtours,,0out_0in,work 3238088,78977,school,1,1,1,1,mandatory,1,3339.0,3339.0,26897,44.0,7.0,14.0,7.0,,,WALK,2.7965374977721846,,,0out_0in,school -52627721,1283602,work,1,1,1,1,mandatory,1,3410.0,3315.0,435012,64.0,8.0,18.0,10.0,,,DRIVEALONEFREE,-0.600552076704748,no_subtours,,0out_1in,work -52638594,1283868,eatout,1,1,1,1,non_mandatory,1,3314.0,3314.0,435278,172.0,18.0,21.0,3.0,,11.417188649277556,WALK,1.465324908192731,,,0out_0in,eatout +52627721,1283602,work,1,1,1,1,mandatory,1,3410.0,3315.0,435012,64.0,8.0,18.0,10.0,,,DRIVEALONEFREE,-0.6005520767047482,no_subtours,,0out_1in,work +52638594,1283868,eatout,1,1,1,1,non_mandatory,1,3314.0,3314.0,435278,172.0,18.0,21.0,3.0,,11.417188649277557,WALK,1.465324908192731,,,0out_0in,eatout 52638611,1283868,maint,1,1,1,1,atwork,1,3176.0,3176.0,435278,154.0,16.0,16.0,0.0,,19.272682099619168,WALK,0.6803915404396673,,52638627.0,0out_0in,atwork -52638627,1283868,work,1,1,1,1,mandatory,1,3176.0,3314.0,435278,79.0,9.0,18.0,9.0,,,DRIVEALONEFREE,-1.2414138285222474,maint,,0out_1in,work -52641825,1283946,work,1,1,1,1,mandatory,1,3619.0,3315.0,435356,48.0,7.0,18.0,11.0,,,TNC_SINGLE,-0.2085333227470863,no_subtours,,0out_0in,work -52668557,1284598,work,1,1,1,1,mandatory,1,3464.0,3261.0,436008,79.0,9.0,18.0,9.0,,,DRIVEALONEFREE,-0.1035644293674547,no_subtours,,0out_0in,work +52638627,1283868,work,1,1,1,1,mandatory,1,3176.0,3314.0,435278,79.0,9.0,18.0,9.0,,,DRIVEALONEFREE,-1.2414138285222476,maint,,0out_1in,work +52641825,1283946,work,1,1,1,1,mandatory,1,3619.0,3315.0,435356,48.0,7.0,18.0,11.0,,,TNC_SINGLE,-0.20853332274708639,no_subtours,,0out_0in,work +52668557,1284598,work,1,1,1,1,mandatory,1,3464.0,3261.0,436008,79.0,9.0,18.0,9.0,,,DRIVEALONEFREE,-0.10356442936745472,no_subtours,,0out_0in,work 52734819,1286215,eat,1,1,1,1,atwork,1,3176.0,3176.0,437625,88.0,10.0,13.0,3.0,,19.30442083196423,WALK,0.6365738436949087,,52734854.0,0out_0in,atwork 52734854,1286215,work,1,1,1,1,mandatory,1,3176.0,3317.0,437625,65.0,8.0,19.0,11.0,,,DRIVEALONEFREE,-1.216142661769996,eat,,0out_2in,work -52897577,1290184,shopping,2,1,1,2,non_mandatory,1,3131.0,3195.0,441594,71.0,9.0,10.0,1.0,,11.507230007796329,SHARED2FREE,-0.2424171251277307,,,0out_0in,shopping -52897578,1290184,shopping,2,2,2,2,non_mandatory,1,3131.0,3195.0,441594,154.0,16.0,16.0,0.0,,11.519310381558869,WALK,-0.2459444422249825,,,0out_0in,shopping +52897577,1290184,shopping,2,1,1,2,non_mandatory,1,3131.0,3195.0,441594,71.0,9.0,10.0,1.0,,11.507230007796327,SHARED2FREE,-0.24241712512773075,,,0out_0in,shopping +52897578,1290184,shopping,2,2,2,2,non_mandatory,1,3131.0,3195.0,441594,154.0,16.0,16.0,0.0,,11.519310381558867,WALK,-0.24594444222498252,,,0out_0in,shopping 52897583,1290184,work,1,1,1,1,mandatory,1,3176.0,3195.0,441594,91.0,10.0,16.0,6.0,,,DRIVEALONEFREE,-0.4742431906622577,no_subtours,,0out_0in,work -52915670,1290626,eat,1,1,1,1,atwork,1,3340.0,3502.0,442036,86.0,10.0,11.0,1.0,,18.513981739804255,BIKE,14.60281456532542,,52915705.0,0out_1in,atwork -52915705,1290626,work,1,1,1,1,mandatory,1,3502.0,3278.0,442036,64.0,8.0,18.0,10.0,,,BIKE,-0.0288128226314909,eat,,0out_2in,work +52915670,1290626,eat,1,1,1,1,atwork,1,3340.0,3502.0,442036,86.0,10.0,11.0,1.0,,18.513981739804258,BIKE,14.60281456532542,,52915705.0,0out_1in,atwork +52915705,1290626,work,1,1,1,1,mandatory,1,3502.0,3278.0,442036,64.0,8.0,18.0,10.0,,,BIKE,-0.028812822631490976,eat,,0out_2in,work 76379130,1862905,othdiscr,1,1,1,1,non_mandatory,1,3460.0,3598.0,721960,151.0,15.0,21.0,6.0,,12.841187222979508,WALK,0.5769914886883216,,,0out_0in,othdiscr 76379171,1862906,othdiscr,1,1,1,1,non_mandatory,1,3705.0,3598.0,721960,104.0,11.0,16.0,5.0,,12.884342308539214,WALK,0.7912388057447584,,,0out_0in,othdiscr -80946571,1974306,othdiscr,1,1,1,1,non_mandatory,1,3176.0,3396.0,760593,59.0,8.0,13.0,5.0,,13.385057255164831,SHARED2FREE,-0.2639203430422295,,,0out_0in,othdiscr -80946591,1974307,eat,1,1,1,1,atwork,1,3574.0,3316.0,760593,113.0,12.0,13.0,1.0,,18.535862390079988,TNC_SINGLE,-1.395417499795803,,80946626.0,0out_0in,atwork -80946626,1974307,work,1,1,1,1,mandatory,1,3316.0,3396.0,760593,78.0,9.0,17.0,8.0,,,SHARED2FREE,-0.1094760736283397,eat,,1out_1in,work -80946637,1974308,escort,1,1,1,1,non_mandatory,1,3315.0,3396.0,760593,0.0,5.0,5.0,0.0,,12.64221480734466,WALK,-0.4993149033005503,,,0out_0in,escort +80946571,1974306,othdiscr,1,1,1,1,non_mandatory,1,3176.0,3396.0,760593,59.0,8.0,13.0,5.0,,13.385057255164831,SHARED2FREE,-0.26392034304222955,,,0out_0in,othdiscr +80946591,1974307,eat,1,1,1,1,atwork,1,3574.0,3316.0,760593,113.0,12.0,13.0,1.0,,18.535862390079988,TNC_SINGLE,-1.3954174997958033,,80946626.0,0out_0in,atwork +80946626,1974307,work,1,1,1,1,mandatory,1,3316.0,3396.0,760593,78.0,9.0,17.0,8.0,,,SHARED2FREE,-0.10947607362833972,eat,,1out_1in,work +80946637,1974308,escort,1,1,1,1,non_mandatory,1,3315.0,3396.0,760593,0.0,5.0,5.0,0.0,,12.64221480734466,WALK,-0.49931490330055034,,,0out_0in,escort 81048440,1976791,escort,1,1,1,1,non_mandatory,1,3314.0,3281.0,761445,124.0,13.0,13.0,0.0,,12.689453354918568,SHARED2FREE,-0.8229389480344289,,,0out_0in,escort 81048476,1976792,eat,1,1,1,1,atwork,1,3176.0,3176.0,761445,85.0,10.0,10.0,0.0,,19.43069515696065,DRIVEALONEFREE,0.6823981998609486,,81048511.0,0out_0in,atwork 81048497,1976792,othdiscr,1,1,1,1,non_mandatory,1,3398.0,3281.0,761445,137.0,14.0,16.0,2.0,,12.916929977944218,DRIVEALONEFREE,0.4546141474460643,,,0out_0in,othdiscr 81048511,1976792,work,1,1,1,1,mandatory,1,3176.0,3281.0,761445,9.0,5.0,14.0,9.0,,,DRIVEALONEFREE,-0.6065542396018551,eat,,0out_0in,work 81130344,1978788,social,1,1,1,1,non_mandatory,1,3498.0,3598.0,762159,66.0,8.0,20.0,12.0,,11.919599153230113,SHARED3FREE,0.5693977967774873,,,0out_0in,social -81130399,1978790,escort,1,1,1,1,non_mandatory,1,3410.0,3598.0,762159,54.0,8.0,8.0,0.0,,12.580660633126431,SHARED2FREE,-0.7876904915091273,,,0out_0in,escort -81130429,1978790,work,1,1,1,1,mandatory,1,3574.0,3598.0,762159,63.0,8.0,17.0,9.0,,,DRIVEALONEFREE,-0.1419369306489097,no_subtours,,0out_0in,work -81130470,1978791,work,1,1,1,1,mandatory,1,3176.0,3598.0,762159,47.0,7.0,17.0,10.0,,,SHARED2FREE,-0.9914397742001736,no_subtours,,0out_0in,work +81130399,1978790,escort,1,1,1,1,non_mandatory,1,3410.0,3598.0,762159,54.0,8.0,8.0,0.0,,12.580660633126433,SHARED2FREE,-0.7876904915091273,,,0out_0in,escort +81130429,1978790,work,1,1,1,1,mandatory,1,3574.0,3598.0,762159,63.0,8.0,17.0,9.0,,,DRIVEALONEFREE,-0.14193693064890978,no_subtours,,0out_0in,work +81130470,1978791,work,1,1,1,1,mandatory,1,3176.0,3598.0,762159,47.0,7.0,17.0,10.0,,,SHARED2FREE,-0.9914397742001735,no_subtours,,0out_0in,work 102419958,2498047,school,1,1,1,1,mandatory,1,3340.0,3383.0,922602,76.0,9.0,15.0,6.0,,,WALK_LRF,2.873022403355501,,,0out_0in,school 102420007,2498048,work,1,1,1,1,mandatory,1,3383.0,3383.0,922602,46.0,7.0,16.0,9.0,,,WALK,2.281344345550102,no_subtours,,0out_0in,work 102420048,2498049,work,1,1,1,1,mandatory,1,3115.0,3383.0,922602,29.0,6.0,16.0,10.0,,,BIKE,0.5927455937959631,no_subtours,,0out_0in,work @@ -65,10 +65,10 @@ tour_id,person_id,tour_type,tour_type_count,tour_type_num,tour_num,tour_count,to 107509922,2622193,escort,1,1,1,2,non_mandatory,1,3376.0,3505.0,952720,19.0,6.0,6.0,0.0,,12.84512392061393,SHARED2FREE,-0.7469201256899531,,,0out_0in,escort 107509941,2622193,othmaint,1,1,2,2,non_mandatory,1,3176.0,3505.0,952720,59.0,8.0,13.0,5.0,,12.428456977574594,DRIVEALONEFREE,-0.748210620713289,,,0out_0in,othmaint 107509987,2622194,shopping,1,1,1,1,non_mandatory,1,3131.0,3505.0,952720,75.0,9.0,14.0,5.0,,11.82505917210944,TNC_SINGLE,-0.6891536748380889,,,0out_0in,shopping -107510034,2622195,work,1,1,1,1,mandatory,1,3363.0,3505.0,952720,63.0,8.0,17.0,9.0,,,DRIVEALONEFREE,0.3860087172282153,no_subtours,,0out_0in,work -116640406,2844887,work,1,1,1,1,mandatory,1,3497.0,3597.0,1028031,107.0,11.0,19.0,8.0,,,DRIVEALONEFREE,0.1107975465014056,no_subtours,,0out_1in,work +107510034,2622195,work,1,1,1,1,mandatory,1,3363.0,3505.0,952720,63.0,8.0,17.0,9.0,,,DRIVEALONEFREE,0.38600871722821534,no_subtours,,0out_0in,work +116640406,2844887,work,1,1,1,1,mandatory,1,3497.0,3597.0,1028031,107.0,11.0,19.0,8.0,,,DRIVEALONEFREE,0.11079754650140568,no_subtours,,0out_1in,work 120287676,2933845,school,1,1,1,1,mandatory,1,3536.0,3134.0,1048898,122.0,12.0,22.0,10.0,,,SHARED3FREE,-1.0938793779365912,,,0out_0in,school 120287717,2933846,school,1,1,1,1,mandatory,1,3536.0,3134.0,1048898,62.0,8.0,16.0,8.0,,,SHARED2FREE,-1.1274402981153004,,,0out_0in,school 120287752,2933847,othdiscr,1,1,1,1,non_mandatory,1,3272.0,3134.0,1048898,42.0,7.0,12.0,5.0,,12.770313936483376,SHARED2FREE,0.5634771825641499,,,0out_1in,othdiscr -120287807,2933848,work,1,1,1,1,mandatory,1,3536.0,3134.0,1048898,31.0,6.0,18.0,12.0,,,DRIVEALONEFREE,0.0371166413339574,no_subtours,,0out_1in,work +120287807,2933848,work,1,1,1,1,mandatory,1,3536.0,3134.0,1048898,31.0,6.0,18.0,12.0,,,DRIVEALONEFREE,0.037116641333957465,no_subtours,,0out_1in,work 131881533,3216622,school,1,1,1,1,mandatory,1,3112.0,3098.0,1148260,138.0,14.0,17.0,3.0,,,WALK,-0.8109781504489971,,,0out_2in,univ diff --git a/activitysim/examples/placeholder_sandag/test/regress/final_1_zone_tours_sh.csv b/activitysim/examples/placeholder_sandag/test/regress/final_1_zone_tours_sh.csv new file mode 100644 index 0000000000..c5f84186bb --- /dev/null +++ b/activitysim/examples/placeholder_sandag/test/regress/final_1_zone_tours_sh.csv @@ -0,0 +1,74 @@ +tour_id,person_id,tour_type,tour_type_count,tour_type_num,tour_num,tour_count,tour_category,number_of_participants,destination,origin,household_id,tdd,start,end,duration,composition,destination_logsum,tour_mode,mode_choice_logsum,atwork_subtour_frequency,parent_tour_id,stop_frequency,primary_purpose +1359025,33146,work,1,1,1,1,mandatory,1,3131.0,3259.0,12593,11.0,5.0,16.0,11.0,,,SHARED2FREE,0.04497099667974337,no_subtours,,2out_0in,work +1359066,33147,work,1,1,1,1,mandatory,1,3342.0,3259.0,12593,59.0,8.0,13.0,5.0,,,SHARED2FREE,0.923483008481964,no_subtours,,0out_3in,work +1494647,36454,shopping,2,1,1,2,non_mandatory,1,3342.0,3134.0,13797,135.0,14.0,14.0,0.0,,11.401050563880887,DRIVEALONEFREE,0.2595703435486286,,,0out_1in,shopping +1494648,36454,shopping,2,2,2,2,non_mandatory,1,3497.0,3134.0,13797,154.0,16.0,16.0,0.0,,11.439449857137804,SHARED2FREE,-0.0006093410329746202,,,1out_0in,shopping +1494694,36455,work,1,1,1,1,mandatory,1,3317.0,3134.0,13797,53.0,7.0,23.0,16.0,,,SHARED3FREE,0.2738771147247297,no_subtours,,0out_0in,work +1709911,41705,eatout,1,1,1,1,non_mandatory,1,3437.0,3342.0,15777,85.0,10.0,10.0,0.0,,11.909980318826895,SHARED2FREE,0.7916769643872005,,,0out_0in,eatout +1709946,41706,business,1,1,1,1,atwork,1,3131.0,3398.0,15777,117.0,12.0,17.0,5.0,,18.502192217204307,SHARED2FREE,-1.0113258602519364,,1709985.0,0out_0in,atwork +1709985,41706,work,1,1,1,1,mandatory,1,3398.0,3342.0,15777,65.0,8.0,19.0,11.0,,,DRIVEALONEFREE,0.5212273251520757,business1,,0out_1in,work +2051456,50035,social,1,1,1,1,joint,2,3497.0,3493.0,18261,146.0,15.0,16.0,1.0,mixed,11.601381369295755,SHARED2FREE,-1.4037659073121553,,,0out_2in,social +2051466,50035,school,1,1,1,1,mandatory,1,3340.0,3493.0,18261,42.0,7.0,12.0,5.0,,,SHARED3FREE,-1.3497203179845243,,,0out_0in,school +2051468,50035,shopping,1,1,1,1,non_mandatory,1,3396.0,3493.0,18261,172.0,18.0,21.0,3.0,,11.366590261069065,DRIVEALONEFREE,0.6130171387405682,,,0out_0in,shopping +2051509,50036,shopping,1,1,1,1,non_mandatory,1,3497.0,3493.0,18261,99.0,11.0,11.0,0.0,,11.555669525382719,DRIVEALONEFREE,0.6647948166112536,,,0out_1in,shopping +2051556,50037,work,1,1,1,1,mandatory,1,3396.0,3493.0,18261,44.0,7.0,14.0,7.0,,,DRIVEALONEFREE,0.5299854110237027,no_subtours,,0out_0in,work +2268889,55338,school,1,1,1,1,mandatory,1,3592.0,3592.0,19758,40.0,7.0,10.0,3.0,,,WALK,0.33183209199757385,,,0out_0in,school +2268938,55339,work,1,1,1,1,mandatory,1,3497.0,3592.0,19758,11.0,5.0,16.0,11.0,,,WALK,0.537772515990635,no_subtours,,0out_0in,work +2373816,57897,work,1,1,1,1,mandatory,1,3176.0,3746.0,20552,50.0,7.0,20.0,13.0,,,DRIVEALONEFREE,-0.28388579338221,no_subtours,,0out_0in,work +2373822,57898,eat,1,1,1,1,atwork,1,3410.0,3347.0,20552,125.0,13.0,14.0,1.0,,11.100624000564997,DRIVEALONEFREE,0.7681389958489114,,2373857.0,0out_0in,atwork +2373857,57898,work,1,1,1,1,mandatory,1,3347.0,3746.0,20552,103.0,11.0,15.0,4.0,,,DRIVEALONEFREE,0.520379765074684,eat,,0out_1in,work +2373898,57899,work,1,1,1,1,mandatory,1,3402.0,3746.0,20552,47.0,7.0,17.0,10.0,,,WALK,1.0388895039783694,no_subtours,,0out_0in,work +2373980,57901,work,2,1,1,2,mandatory,1,3115.0,3746.0,20552,25.0,6.0,12.0,6.0,,,SHARED3FREE,0.6022315390131013,no_subtours,,0out_0in,work +2373981,57901,work,2,2,2,2,mandatory,1,3115.0,3746.0,20552,150.0,15.0,20.0,5.0,,,SHARED2FREE,0.6232767878249469,no_subtours,,1out_0in,work +2563802,62531,school,1,1,1,1,mandatory,1,3460.0,3316.0,21869,180.0,20.0,20.0,0.0,,,SHARED3FREE,-0.7094603590463964,,,0out_0in,school +2563821,62532,escort,1,1,1,1,non_mandatory,1,3398.0,3316.0,21869,20.0,6.0,7.0,1.0,,12.499268454965652,SHARED2FREE,-1.4604154628072699,,,0out_0in,escort +2563862,62533,escort,3,1,1,4,non_mandatory,1,3402.0,3316.0,21869,1.0,5.0,6.0,1.0,,12.534424209198946,SHARED3FREE,-1.2940574569954848,,,0out_3in,escort +2563863,62533,escort,3,2,2,4,non_mandatory,1,3519.0,3316.0,21869,99.0,11.0,11.0,0.0,,12.466623656700463,SHARED2FREE,-0.9326373013150777,,,0out_0in,escort +2563864,62533,escort,3,3,3,4,non_mandatory,1,3398.0,3316.0,21869,135.0,14.0,14.0,0.0,,12.485628371830364,SHARED3FREE,-1.4467093822325976,,,0out_0in,escort +2563878,62533,othdiscr,1,1,4,4,non_mandatory,1,3383.0,3316.0,21869,99.0,11.0,11.0,0.0,,12.893824062601642,DRIVEALONEFREE,0.09001472929003788,,,0out_0in,othdiscr +2563925,62534,school,1,1,1,1,mandatory,1,3316.0,3316.0,21869,55.0,8.0,9.0,1.0,,,SHARED3FREE,0.4282867689528016,,,0out_1in,school +2787968,67999,escort,1,1,1,2,non_mandatory,1,3410.0,3378.0,23619,124.0,13.0,13.0,0.0,,12.955709049606325,TNC_SINGLE,-0.5513480052414359,,,0out_2in,escort +2787995,67999,social,1,1,2,2,non_mandatory,1,3400.0,3378.0,23619,165.0,17.0,20.0,3.0,,11.998752822352907,WALK,0.8829112299755879,,,0out_0in,social +2788039,68000,work,1,1,1,1,mandatory,1,3375.0,3378.0,23619,50.0,7.0,20.0,13.0,,,WALK,0.6308240193619252,no_subtours,,0out_0in,work +3238088,78977,school,1,1,1,1,mandatory,1,3339.0,3339.0,26897,44.0,7.0,14.0,7.0,,,WALK,2.7965374977721846,,,0out_0in,school +52627721,1283602,work,1,1,1,1,mandatory,1,3410.0,3315.0,435012,64.0,8.0,18.0,10.0,,,DRIVEALONEFREE,-0.6005520767047482,no_subtours,,0out_1in,work +52638594,1283868,eatout,1,1,1,1,non_mandatory,1,3314.0,3314.0,435278,172.0,18.0,21.0,3.0,,11.417188649277557,WALK,1.465324908192731,,,0out_0in,eatout +52638611,1283868,maint,1,1,1,1,atwork,1,3176.0,3176.0,435278,154.0,16.0,16.0,0.0,,19.272682099619168,WALK,0.6803915404396673,,52638627.0,0out_0in,atwork +52638627,1283868,work,1,1,1,1,mandatory,1,3176.0,3314.0,435278,79.0,9.0,18.0,9.0,,,DRIVEALONEFREE,-1.2414138285222476,maint,,0out_1in,work +52641825,1283946,work,1,1,1,1,mandatory,1,3619.0,3315.0,435356,48.0,7.0,18.0,11.0,,,TNC_SINGLE,-0.20853332274708639,no_subtours,,0out_0in,work +52668557,1284598,work,1,1,1,1,mandatory,1,3464.0,3261.0,436008,79.0,9.0,18.0,9.0,,,DRIVEALONEFREE,-0.10356442936745472,no_subtours,,0out_0in,work +52734819,1286215,eat,1,1,1,1,atwork,1,3176.0,3176.0,437625,88.0,10.0,13.0,3.0,,19.30442083196423,WALK,0.6365738436949087,,52734854.0,0out_0in,atwork +52734854,1286215,work,1,1,1,1,mandatory,1,3176.0,3317.0,437625,65.0,8.0,19.0,11.0,,,DRIVEALONEFREE,-1.216142661769996,eat,,0out_2in,work +52897577,1290184,shopping,2,1,1,2,non_mandatory,1,3131.0,3195.0,441594,71.0,9.0,10.0,1.0,,11.507230007796327,SHARED2FREE,-0.24241712512773075,,,0out_0in,shopping +52897578,1290184,shopping,2,2,2,2,non_mandatory,1,3131.0,3195.0,441594,154.0,16.0,16.0,0.0,,11.519310381558867,WALK,-0.24594444222498252,,,0out_0in,shopping +52897583,1290184,work,1,1,1,1,mandatory,1,3176.0,3195.0,441594,91.0,10.0,16.0,6.0,,,DRIVEALONEFREE,-0.4742431906622577,no_subtours,,0out_0in,work +52915670,1290626,eat,1,1,1,1,atwork,1,3340.0,3502.0,442036,86.0,10.0,11.0,1.0,,18.513981739804258,BIKE,14.60281456532542,,52915705.0,0out_1in,atwork +52915705,1290626,work,1,1,1,1,mandatory,1,3502.0,3278.0,442036,64.0,8.0,18.0,10.0,,,BIKE,-0.028812822631490976,eat,,0out_2in,work +76379130,1862905,othdiscr,1,1,1,1,non_mandatory,1,3460.0,3598.0,721960,151.0,15.0,21.0,6.0,,12.841187222979508,WALK,0.5769914886883216,,,0out_0in,othdiscr +76379171,1862906,othdiscr,1,1,1,1,non_mandatory,1,3705.0,3598.0,721960,104.0,11.0,16.0,5.0,,12.884342308539214,WALK,0.7912388057447584,,,0out_0in,othdiscr +80946571,1974306,othdiscr,1,1,1,1,non_mandatory,1,3176.0,3396.0,760593,59.0,8.0,13.0,5.0,,13.385057255164831,SHARED2FREE,-0.26392034304222955,,,0out_0in,othdiscr +80946591,1974307,eat,1,1,1,1,atwork,1,3574.0,3316.0,760593,113.0,12.0,13.0,1.0,,18.535862390079988,TNC_SINGLE,-1.3954174997958033,,80946626.0,0out_0in,atwork +80946626,1974307,work,1,1,1,1,mandatory,1,3316.0,3396.0,760593,78.0,9.0,17.0,8.0,,,SHARED2FREE,-0.10947607362833972,eat,,1out_1in,work +80946637,1974308,escort,1,1,1,1,non_mandatory,1,3315.0,3396.0,760593,0.0,5.0,5.0,0.0,,12.64221480734466,WALK,-0.49931490330055034,,,0out_0in,escort +81048440,1976791,escort,1,1,1,1,non_mandatory,1,3314.0,3281.0,761445,124.0,13.0,13.0,0.0,,12.689453354918568,SHARED2FREE,-0.8229389480344289,,,0out_0in,escort +81048476,1976792,eat,1,1,1,1,atwork,1,3176.0,3176.0,761445,85.0,10.0,10.0,0.0,,19.43069515696065,DRIVEALONEFREE,0.6823981998609486,,81048511.0,0out_0in,atwork +81048497,1976792,othdiscr,1,1,1,1,non_mandatory,1,3398.0,3281.0,761445,137.0,14.0,16.0,2.0,,12.916929977944218,DRIVEALONEFREE,0.4546141474460643,,,0out_0in,othdiscr +81048511,1976792,work,1,1,1,1,mandatory,1,3176.0,3281.0,761445,9.0,5.0,14.0,9.0,,,DRIVEALONEFREE,-0.6065542396018551,eat,,0out_0in,work +81130344,1978788,social,1,1,1,1,non_mandatory,1,3498.0,3598.0,762159,66.0,8.0,20.0,12.0,,11.919599153230113,SHARED3FREE,0.5693977967774873,,,0out_0in,social +81130399,1978790,escort,1,1,1,1,non_mandatory,1,3410.0,3598.0,762159,54.0,8.0,8.0,0.0,,12.580660633126433,SHARED2FREE,-0.7876904915091273,,,0out_0in,escort +81130429,1978790,work,1,1,1,1,mandatory,1,3574.0,3598.0,762159,63.0,8.0,17.0,9.0,,,DRIVEALONEFREE,-0.14193693064890978,no_subtours,,0out_0in,work +81130470,1978791,work,1,1,1,1,mandatory,1,3176.0,3598.0,762159,47.0,7.0,17.0,10.0,,,SHARED2FREE,-0.9914397742001735,no_subtours,,0out_0in,work +102419958,2498047,school,1,1,1,1,mandatory,1,3340.0,3383.0,922602,76.0,9.0,15.0,6.0,,,WALK_LRF,2.873022403355501,,,0out_0in,school +102420007,2498048,work,1,1,1,1,mandatory,1,3383.0,3383.0,922602,46.0,7.0,16.0,9.0,,,WALK,2.281344345550102,no_subtours,,0out_0in,work +102420048,2498049,work,1,1,1,1,mandatory,1,3115.0,3383.0,922602,29.0,6.0,16.0,10.0,,,BIKE,0.5927455937959631,no_subtours,,0out_0in,work +107509903,2622192,school,1,1,1,1,mandatory,1,3378.0,3505.0,952720,44.0,7.0,14.0,7.0,,,WALK,-0.2639461480591949,,,0out_0in,school +107509922,2622193,escort,1,1,1,2,non_mandatory,1,3376.0,3505.0,952720,19.0,6.0,6.0,0.0,,12.84512392061393,SHARED2FREE,-0.7469201256899531,,,0out_0in,escort +107509941,2622193,othmaint,1,1,2,2,non_mandatory,1,3176.0,3505.0,952720,59.0,8.0,13.0,5.0,,12.428456977574594,DRIVEALONEFREE,-0.748210620713289,,,0out_0in,othmaint +107509987,2622194,shopping,1,1,1,1,non_mandatory,1,3131.0,3505.0,952720,75.0,9.0,14.0,5.0,,11.82505917210944,TNC_SINGLE,-0.6891536748380889,,,0out_0in,shopping +107510034,2622195,work,1,1,1,1,mandatory,1,3363.0,3505.0,952720,63.0,8.0,17.0,9.0,,,DRIVEALONEFREE,0.38600871722821534,no_subtours,,0out_0in,work +116640406,2844887,work,1,1,1,1,mandatory,1,3497.0,3597.0,1028031,107.0,11.0,19.0,8.0,,,DRIVEALONEFREE,0.11079754650140568,no_subtours,,0out_1in,work +120287676,2933845,school,1,1,1,1,mandatory,1,3536.0,3134.0,1048898,122.0,12.0,22.0,10.0,,,SHARED3FREE,-1.0938793779365912,,,0out_0in,school +120287717,2933846,school,1,1,1,1,mandatory,1,3536.0,3134.0,1048898,62.0,8.0,16.0,8.0,,,SHARED2FREE,-1.1274402981153004,,,0out_0in,school +120287752,2933847,othdiscr,1,1,1,1,non_mandatory,1,3272.0,3134.0,1048898,42.0,7.0,12.0,5.0,,12.770313936483376,SHARED2FREE,0.5634771825641499,,,0out_1in,othdiscr +120287807,2933848,work,1,1,1,1,mandatory,1,3536.0,3134.0,1048898,31.0,6.0,18.0,12.0,,,DRIVEALONEFREE,0.037116641333957465,no_subtours,,0out_1in,work +131881533,3216622,school,1,1,1,1,mandatory,1,3112.0,3098.0,1148260,138.0,14.0,17.0,3.0,,,WALK,-0.8109781504489971,,,0out_2in,univ diff --git a/activitysim/examples/placeholder_sandag/test/regress/final_1_zone_trips.csv b/activitysim/examples/placeholder_sandag/test/regress/final_1_zone_trips.csv index f66d5e1aae..3e4eecce69 100644 --- a/activitysim/examples/placeholder_sandag/test/regress/final_1_zone_trips.csv +++ b/activitysim/examples/placeholder_sandag/test/regress/final_1_zone_trips.csv @@ -1,17 +1,17 @@ trip_id,person_id,household_id,primary_purpose,trip_num,outbound,trip_count,destination,origin,tour_id,purpose,destination_logsum,depart,trip_mode,mode_choice_logsum -10872201,33146,12593,work,1,True,3,3290,3259,1359025,escort,12.50569587831954,5,WALK,0.8796640848644649 +10872201,33146,12593,work,1,True,3,3290,3259,1359025,escort,12.505695878319539,5,WALK,0.8796640848644649 10872202,33146,12593,work,2,True,3,3317,3290,1359025,escort,12.74437072347788,7,WALK,0.7528069371143918 -10872203,33146,12593,work,3,True,3,3131,3317,1359025,work,,7,SHARED2FREE,-0.2771584910567812 +10872203,33146,12593,work,3,True,3,3131,3317,1359025,work,,7,SHARED2FREE,-0.27715849105678125 10872205,33146,12593,work,1,False,1,3259,3131,1359025,home,,16,SHARED2FREE,-0.3682899649069351 10872529,33147,12593,work,1,True,1,3342,3259,1359066,work,,8,SHARED2FREE,0.4187992443742946 10872533,33147,12593,work,1,False,4,3383,3342,1359066,escort,14.576775128994838,12,SHARED2FREE,0.8686143194682551 10872534,33147,12593,work,2,False,4,3274,3383,1359066,social,13.66341661662671,13,SHARED2FREE,0.5107943560451084 10872535,33147,12593,work,3,False,4,3340,3274,1359066,escort,14.112546915053883,13,SHARED2FREE,0.4230067105850933 -10872536,33147,12593,work,4,False,4,3259,3340,1359066,home,,13,DRIVEALONEFREE,0.2491163354013192 -11957177,36454,13797,shopping,1,True,1,3342,3134,1494647,shopping,,14,DRIVEALONEFREE,0.2941685907193185 -11957181,36454,13797,shopping,1,False,2,3574,3342,1494647,shopping,12.641799100811618,14,TAXI,-0.1089902633576625 -11957182,36454,13797,shopping,2,False,2,3134,3574,1494647,home,,14,TNC_SINGLE,-0.1841764445667141 -11957185,36454,13797,shopping,1,True,2,3402,3134,1494648,escort,17.45390698273753,16,DRIVEALONEFREE,1.279084913174844 +10872536,33147,12593,work,4,False,4,3259,3340,1359066,home,,13,DRIVEALONEFREE,0.24911633540131922 +11957177,36454,13797,shopping,1,True,1,3342,3134,1494647,shopping,,14,DRIVEALONEFREE,0.29416859071931833 +11957181,36454,13797,shopping,1,False,2,3574,3342,1494647,shopping,12.641799100811618,14,TAXI,-0.10899026335766256 +11957182,36454,13797,shopping,2,False,2,3134,3574,1494647,home,,14,TNC_SINGLE,-0.18417644456671423 +11957185,36454,13797,shopping,1,True,2,3402,3134,1494648,escort,17.45390698273753,16,DRIVEALONEFREE,1.2790849131748439 11957186,36454,13797,shopping,2,True,2,3497,3402,1494648,shopping,,16,SHARED2FREE,1.520377820514991 11957189,36454,13797,shopping,1,False,1,3134,3497,1494648,home,,16,SHARED2FREE,1.253344921201897 11957553,36455,13797,work,1,True,1,3317,3134,1494694,work,,7,WALK,0.5467805012552107 @@ -22,119 +22,119 @@ trip_id,person_id,household_id,primary_purpose,trip_num,outbound,trip_count,dest 13679573,41706,15777,atwork,1,False,1,3398,3131,1709946,work,,17,SHARED2FREE,1.565177581330629 13679881,41706,15777,work,1,True,1,3398,3342,1709985,work,,8,DRIVEALONEFREE,0.229309200564582 13679885,41706,15777,work,1,False,2,3317,3398,1709985,escort,12.99906661478402,18,DRIVEALONEFREE,0.3635288368935579 -13679886,41706,15777,work,2,False,2,3342,3317,1709985,home,,19,WALK,0.1992038827303373 +13679886,41706,15777,work,2,False,2,3342,3317,1709985,home,,19,WALK,0.19920388273033734 16411649,50035,18261,social,1,True,1,3497,3493,2051456,social,,15,SHARED2FREE,0.8144179602536189 -16411653,50035,18261,social,1,False,3,3398,3497,2051456,social,13.607997193750748,16,SHARED2FREE,0.9055182726514672 -16411654,50035,18261,social,2,False,3,3259,3398,2051456,social,14.227902019942242,16,SHARED2FREE,0.4816403510255787 +16411653,50035,18261,social,1,False,3,3398,3497,2051456,social,13.607997193750748,16,SHARED2FREE,0.9055182726514673 +16411654,50035,18261,social,2,False,3,3259,3398,2051456,social,14.227902019942242,16,SHARED2FREE,0.48164035102557873 16411655,50035,18261,social,3,False,3,3493,3259,2051456,home,,16,TAXI,0.4624895573096613 16411729,50035,18261,school,1,True,1,3340,3493,2051466,school,,7,SHARED3FREE,-0.3581378319265033 16411733,50035,18261,school,1,False,1,3493,3340,2051466,home,,12,DRIVEALONEFREE,-0.7018451340834048 16411745,50035,18261,shopping,1,True,1,3396,3493,2051468,shopping,,18,DRIVEALONEFREE,0.2675492794224116 -16411749,50035,18261,shopping,1,False,1,3493,3396,2051468,home,,21,DRIVEALONEFREE,0.2641141398041625 +16411749,50035,18261,shopping,1,False,1,3493,3396,2051468,home,,21,DRIVEALONEFREE,0.26411413980416254 16412073,50036,18261,shopping,1,True,1,3497,3493,2051509,shopping,,11,WALK,0.3815001484705561 -16412077,50036,18261,shopping,1,False,2,3378,3497,2051509,escort,12.868283363745013,11,DRIVEALONEFREE,-0.1609182250241023 +16412077,50036,18261,shopping,1,False,2,3378,3497,2051509,escort,12.868283363745013,11,DRIVEALONEFREE,-0.16091822502410236 16412078,50036,18261,shopping,2,False,2,3493,3378,2051509,home,,11,DRIVEALONEFREE,-0.2842450511223511 16412449,50037,18261,work,1,True,1,3396,3493,2051556,work,,7,DRIVEALONEFREE,0.2543385882925606 16412453,50037,18261,work,1,False,1,3493,3396,2051556,home,,14,DRIVEALONEFREE,0.2544109368585356 18151113,55338,19758,school,1,True,1,3592,3592,2268889,school,,7,WALK,-0.5037709043643227 18151117,55338,19758,school,1,False,1,3592,3592,2268889,home,,10,WALK,-0.5037709043643227 -18151505,55339,19758,work,1,True,1,3497,3592,2268938,work,,5,WALK,-0.0391673467852052 -18151509,55339,19758,work,1,False,1,3592,3497,2268938,home,,16,WALK,-0.0391618013350057 +18151505,55339,19758,work,1,True,1,3497,3592,2268938,work,,5,WALK,-0.03916734678520527 +18151509,55339,19758,work,1,False,1,3592,3497,2268938,home,,16,WALK,-0.039161801335005776 18990529,57897,20552,work,1,True,1,3176,3746,2373816,work,,7,DRIVEALONEFREE,-0.8320971690146864 -18990533,57897,20552,work,1,False,1,3746,3176,2373816,home,,20,DRIVEALONEFREE,-1.037987421384564 -18990577,57898,20552,atwork,1,True,1,3410,3347,2373822,atwork,,13,DRIVEALONEFREE,0.1572164518377817 -18990581,57898,20552,atwork,1,False,1,3347,3410,2373822,work,,14,DRIVEALONEFREE,0.1565616339228507 +18990533,57897,20552,work,1,False,1,3746,3176,2373816,home,,20,DRIVEALONEFREE,-1.0379874213845641 +18990577,57898,20552,atwork,1,True,1,3410,3347,2373822,atwork,,13,DRIVEALONEFREE,0.15721645183778177 +18990581,57898,20552,atwork,1,False,1,3347,3410,2373822,work,,14,DRIVEALONEFREE,0.15656163392285075 18990857,57898,20552,work,1,True,1,3347,3746,2373857,work,,11,DRIVEALONEFREE,-0.2916992537280374 18990861,57898,20552,work,1,False,2,3849,3347,2373857,escort,11.83577521389007,15,DRIVEALONEFREE,-0.2521012321385745 18990862,57898,20552,work,2,False,2,3746,3849,2373857,home,,15,DRIVEALONEFREE,-0.0911874930860481 18991185,57899,20552,work,1,True,1,3402,3746,2373898,work,,7,WALK,-1.1548338964177454 18991189,57899,20552,work,1,False,1,3746,3402,2373898,home,,17,WALK,-1.229178185082696 -18991841,57901,20552,work,1,True,1,3115,3746,2373980,work,,6,DRIVEALONEFREE,0.1957407771020773 -18991845,57901,20552,work,1,False,1,3746,3115,2373980,home,,11,DRIVEALONEFREE,0.2077940476290929 -18991849,57901,20552,work,1,True,2,3460,3746,2373981,othmaint,12.489995670716882,15,SHARED2FREE,0.249460922022029 -18991850,57901,20552,work,2,True,2,3115,3460,2373981,work,,16,DRIVEALONEFREE,0.1059704675141837 -18991853,57901,20552,work,1,False,1,3746,3115,2373981,home,,18,SHARED2FREE,0.2208684199770616 +18991841,57901,20552,work,1,True,1,3115,3746,2373980,work,,6,DRIVEALONEFREE,0.19574077710207732 +18991845,57901,20552,work,1,False,1,3746,3115,2373980,home,,12,DRIVEALONEFREE,0.20779404762909298 +18991849,57901,20552,work,1,True,2,3460,3746,2373981,othmaint,12.489995670716882,15,SHARED2FREE,0.24946092202202907 +18991850,57901,20552,work,2,True,2,3115,3460,2373981,work,,16,DRIVEALONEFREE,0.10597046751418379 +18991853,57901,20552,work,1,False,1,3746,3115,2373981,home,,20,SHARED2FREE,0.23660752783217825 20510417,62531,21869,school,1,True,1,3460,3316,2563802,school,,20,SHARED3FREE,-1.4448137456466916 20510421,62531,21869,school,1,False,1,3316,3460,2563802,home,,20,WALK,-1.5207459403958272 -20510569,62532,21869,escort,1,True,1,3398,3316,2563821,escort,,6,SHARED2FREE,0.1786959845402289 -20510573,62532,21869,escort,1,False,1,3316,3398,2563821,home,,7,DRIVEALONEFREE,0.2004514945825397 +20510569,62532,21869,escort,1,True,1,3398,3316,2563821,escort,,6,SHARED2FREE,0.17869598454022895 +20510573,62532,21869,escort,1,False,1,3316,3398,2563821,home,,7,DRIVEALONEFREE,0.20045149458253975 20510897,62533,21869,escort,1,True,1,3402,3316,2563862,escort,,5,SHARED3FREE,0.7112775892674524 20510901,62533,21869,escort,1,False,4,3339,3402,2563862,shopping,14.091505819104302,6,SHARED3FREE,0.983638157033812 -20510902,62533,21869,escort,2,False,4,3505,3339,2563862,eatout,15.083442995634533,6,SHARED3FREE,0.9900387210738842 -20510903,62533,21869,escort,3,False,4,3536,3505,2563862,escort,15.770594264076768,6,SHARED2FREE,0.7686807718987421 +20510902,62533,21869,escort,2,False,4,3505,3339,2563862,eatout,15.083442995634531,6,SHARED3FREE,0.9900387210738842 +20510903,62533,21869,escort,3,False,4,3536,3505,2563862,escort,15.770594264076767,6,SHARED2FREE,0.7686807718987421 20510904,62533,21869,escort,4,False,4,3316,3536,2563862,home,,6,SHARED3FREE,0.5576305299679216 20510905,62533,21869,escort,1,True,1,3519,3316,2563863,escort,,11,SHARED2FREE,0.4802732339996155 -20510909,62533,21869,escort,1,False,1,3316,3519,2563863,home,,11,SHARED2FREE,0.4817271945038804 +20510909,62533,21869,escort,1,False,1,3316,3519,2563863,home,,11,SHARED2FREE,0.48172719450388046 20510913,62533,21869,escort,1,True,1,3398,3316,2563864,escort,,14,SHARED2FREE,0.5739525286758124 20510917,62533,21869,escort,1,False,1,3316,3398,2563864,home,,14,DRIVEALONEFREE,0.5863871055816903 -20511025,62533,21869,othdiscr,1,True,1,3383,3316,2563878,othdiscr,,11,DRIVEALONEFREE,-0.0308449442047451 -20511029,62533,21869,othdiscr,1,False,1,3316,3383,2563878,home,,11,DRIVEALONEFREE,-0.0416324320685514 +20511025,62533,21869,othdiscr,1,True,1,3383,3316,2563878,othdiscr,,11,DRIVEALONEFREE,-0.03084494420474515 +20511029,62533,21869,othdiscr,1,False,1,3316,3383,2563878,home,,11,DRIVEALONEFREE,-0.041632432068551585 20511401,62534,21869,school,1,True,1,3316,3316,2563925,school,,8,WALK,-0.7509920520073805 20511405,62534,21869,school,1,False,2,3363,3316,2563925,shopping,5.114003115539301,9,SHARED3FREE,-1.3212813014651676 20511406,62534,21869,school,2,False,2,3316,3363,2563925,home,,9,WALK,-1.2832199854288413 -22303745,67999,23619,escort,1,True,1,3410,3378,2787968,escort,,13,TNC_SHARED,0.2370003254306411 -22303749,67999,23619,escort,1,False,3,3822,3410,2787968,eatout,12.793892361239552,13,TNC_SINGLE,-0.0219489014987906 -22303750,67999,23619,escort,2,False,3,3412,3822,2787968,escort,12.238309717426684,13,TNC_SINGLE,-0.1578164617938099 -22303751,67999,23619,escort,3,False,3,3378,3412,2787968,home,,13,TNC_SINGLE,0.1106008495770981 -22303961,67999,23619,social,1,True,1,3400,3378,2787995,social,,17,WALK,0.0219271318180106 -22303965,67999,23619,social,1,False,1,3378,3400,2787995,home,,20,WALK,0.0219421437437839 -22304313,68000,23619,work,1,True,1,3375,3378,2788039,work,,7,WALK,0.1634844777978781 -22304317,68000,23619,work,1,False,1,3378,3375,2788039,home,,20,WALK,0.1634844777978781 +22303745,67999,23619,escort,1,True,1,3410,3378,2787968,escort,,13,TNC_SHARED,0.23700032543064115 +22303749,67999,23619,escort,1,False,3,3822,3410,2787968,eatout,12.793892361239553,13,TNC_SINGLE,-0.021948901498790654 +22303750,67999,23619,escort,2,False,3,3412,3822,2787968,escort,12.238309717426683,13,TNC_SINGLE,-0.1578164617938098 +22303751,67999,23619,escort,3,False,3,3378,3412,2787968,home,,13,TNC_SINGLE,0.11060084957709816 +22303961,67999,23619,social,1,True,1,3400,3378,2787995,social,,17,WALK,0.02192713181801061 +22303965,67999,23619,social,1,False,1,3378,3400,2787995,home,,20,WALK,0.021942143743783993 +22304313,68000,23619,work,1,True,1,3375,3378,2788039,work,,7,WALK,0.16348447779787817 +22304317,68000,23619,work,1,False,1,3378,3375,2788039,home,,20,WALK,0.16348447779787817 25904705,78977,26897,school,1,True,1,3339,3339,3238088,school,,7,WALK,0.3224231573284306 25904709,78977,26897,school,1,False,1,3339,3339,3238088,home,,14,WALK,0.3224234088552417 421021769,1283602,435012,work,1,True,1,3410,3315,52627721,work,,8,DRIVEALONEFREE,-0.7004813704631009 421021773,1283602,435012,work,1,False,2,3314,3410,52627721,social,10.399929897293942,14,DRIVEALONEFREE,-0.7505287871501307 421021774,1283602,435012,work,2,False,2,3315,3314,52627721,home,,18,DRIVEALONEFREE,0.1928245919289156 -421108753,1283868,435278,eatout,1,True,1,3314,3314,52638594,eatout,,18,WALK,0.2408265608428425 -421108757,1283868,435278,eatout,1,False,1,3314,3314,52638594,home,,21,WALK,0.2408269372508094 -421108889,1283868,435278,atwork,1,True,1,3176,3176,52638611,atwork,,16,WALK,4.41006666595954 -421108893,1283868,435278,atwork,1,False,1,3176,3176,52638611,work,,16,WALK,4.41006666595954 +421108753,1283868,435278,eatout,1,True,1,3314,3314,52638594,eatout,,18,WALK,0.24082656084284254 +421108757,1283868,435278,eatout,1,False,1,3314,3314,52638594,home,,21,WALK,0.24082693725080942 +421108889,1283868,435278,atwork,1,True,1,3176,3176,52638611,atwork,,16,WALK,4.4100666659595396 +421108893,1283868,435278,atwork,1,False,1,3176,3176,52638611,work,,16,WALK,4.4100666659595396 421109017,1283868,435278,work,1,True,1,3176,3314,52638627,work,,9,DRIVEALONEFREE,-0.8523980347217911 421109021,1283868,435278,work,1,False,2,3176,3176,52638627,work,14.061231380263717,18,WALK,2.2993936628468057 421109022,1283868,435278,work,2,False,2,3314,3176,52638627,home,,18,DRIVEALONEFREE,-0.8993747376742869 421134601,1283946,435356,work,1,True,1,3619,3315,52641825,work,,7,TNC_SINGLE,0.3084730568128902 -421134605,1283946,435356,work,1,False,1,3315,3619,52641825,home,,18,TNC_SINGLE,0.1688496731153186 -421348457,1284598,436008,work,1,True,1,3464,3261,52668557,work,,9,DRIVEALONEFREE,-0.199055801411404 -421348461,1284598,436008,work,1,False,1,3261,3464,52668557,home,,18,DRIVEALONEFREE,-0.1899002521579137 +421134605,1283946,435356,work,1,False,1,3315,3619,52641825,home,,18,TNC_SINGLE,0.16884967311531862 +421348457,1284598,436008,work,1,True,1,3464,3261,52668557,work,,9,DRIVEALONEFREE,-0.19905580141140403 +421348461,1284598,436008,work,1,False,1,3261,3464,52668557,home,,18,DRIVEALONEFREE,-0.18990025215791376 421878553,1286215,437625,atwork,1,True,1,3176,3176,52734819,atwork,,10,WALK,4.410066095224354 421878557,1286215,437625,atwork,1,False,1,3176,3176,52734819,work,,13,WALK,4.410066095224354 421878833,1286215,437625,work,1,True,1,3176,3317,52734854,work,,8,DRIVEALONEFREE,-0.7732401280141026 421878837,1286215,437625,work,1,False,3,3398,3176,52734854,othmaint,13.419982162114987,17,DRIVEALONEFREE,-1.237602447505927 -421878838,1286215,437625,work,2,False,3,3397,3398,52734854,othmaint,12.007055480943816,19,DRIVEALONEFREE,0.3283001989153217 -421878839,1286215,437625,work,3,False,3,3317,3397,52734854,home,,19,WALK,0.1924420139714097 +421878838,1286215,437625,work,2,False,3,3397,3398,52734854,othmaint,12.007055480943817,19,DRIVEALONEFREE,0.32830019891532175 +421878839,1286215,437625,work,3,False,3,3317,3397,52734854,home,,19,WALK,0.19244201397140973 423180617,1290184,441594,shopping,1,True,1,3131,3195,52897577,shopping,,9,SHARED2FREE,0.2504977565051515 -423180621,1290184,441594,shopping,1,False,1,3195,3131,52897577,home,,10,SHARED2FREE,0.2324245999832967 -423180625,1290184,441594,shopping,1,True,1,3131,3195,52897578,shopping,,16,WALK,-1.798118965544273 +423180621,1290184,441594,shopping,1,False,1,3195,3131,52897577,home,,10,SHARED2FREE,0.23242459998329676 +423180625,1290184,441594,shopping,1,True,1,3131,3195,52897578,shopping,,16,WALK,-1.7981189655442733 423180629,1290184,441594,shopping,1,False,1,3195,3131,52897578,home,,16,WALK,-1.891099527786877 423180665,1290184,441594,work,1,True,1,3176,3195,52897583,work,,10,WALK,0.855151926514014 -423180669,1290184,441594,work,1,False,1,3195,3176,52897583,home,,16,WALK,0.9414955852678496 -423325361,1290626,442036,atwork,1,True,1,3340,3502,52915670,atwork,,10,BIKE,0.1045867418182057 +423180669,1290184,441594,work,1,False,1,3195,3176,52897583,home,,16,WALK,0.9414955852678495 +423325361,1290626,442036,atwork,1,True,1,3340,3502,52915670,atwork,,10,BIKE,0.10458674181820576 423325365,1290626,442036,atwork,1,False,2,3176,3340,52915670,othmaint,16.13095576158003,11,BIKE,-0.524564641781727 -423325366,1290626,442036,atwork,2,False,2,3502,3176,52915670,work,,11,BIKE,3.276825955580493 -423325641,1290626,442036,work,1,True,1,3502,3278,52915705,work,,8,BIKE,-1.0096302214124555 +423325366,1290626,442036,atwork,2,False,2,3502,3176,52915670,work,,11,BIKE,3.2768259555804935 +423325641,1290626,442036,work,1,True,1,3502,3278,52915705,work,,8,BIKE,-1.0096302214124557 423325645,1290626,442036,work,1,False,3,3176,3502,52915705,othdiscr,13.17876515323236,18,BIKE,-1.546730219362895 423325646,1290626,442036,work,2,False,3,3176,3176,52915705,othmaint,21.761313694214984,18,WALK,3.1160978512473827 423325647,1290626,442036,work,3,False,3,3278,3176,52915705,home,,18,BIKE,2.471064663969691 -611033041,1862905,721960,othdiscr,1,True,1,3460,3598,76379130,othdiscr,,15,WALK,-1.2345810242068171 +611033041,1862905,721960,othdiscr,1,True,1,3460,3598,76379130,othdiscr,,15,WALK,-1.234581024206817 611033045,1862905,721960,othdiscr,1,False,1,3598,3460,76379130,home,,21,WALK,-0.9300094952392008 611033369,1862906,721960,othdiscr,1,True,1,3705,3598,76379171,othdiscr,,11,WALK,-0.8123168897915816 611033373,1862906,721960,othdiscr,1,False,1,3598,3705,76379171,home,,16,WALK,-0.6995691553182372 647572569,1974306,760593,othdiscr,1,True,1,3176,3396,80946571,othdiscr,,8,SHARED2FREE,-0.2717262829172214 -647572573,1974306,760593,othdiscr,1,False,1,3396,3176,80946571,home,,13,SHARED2FREE,-0.3489570743749159 -647572729,1974307,760593,atwork,1,True,1,3574,3316,80946591,atwork,,12,TNC_SINGLE,0.4703651261640751 -647572733,1974307,760593,atwork,1,False,1,3316,3574,80946591,work,,13,TNC_SINGLE,0.3277430310717987 +647572573,1974306,760593,othdiscr,1,False,1,3396,3176,80946571,home,,13,SHARED2FREE,-0.3489570743749156 +647572729,1974307,760593,atwork,1,True,1,3574,3316,80946591,atwork,,12,TNC_SINGLE,0.47036512616407533 +647572733,1974307,760593,atwork,1,False,1,3316,3574,80946591,work,,13,TNC_SINGLE,0.32774303107179875 647573009,1974307,760593,work,1,True,2,3400,3396,80946626,shopping,12.693917677139323,9,SHARED2FREE,0.8051982291809634 647573010,1974307,760593,work,2,True,2,3316,3400,80946626,work,,10,SHARED2FREE,0.3488831452765107 -647573013,1974307,760593,work,1,False,2,3347,3316,80946626,work,14.620790396011603,16,WALK,0.9232328252322722 -647573014,1974307,760593,work,2,False,2,3396,3347,80946626,home,,17,DRIVEALONEFREE,0.2236952870492791 -647573097,1974308,760593,escort,1,True,1,3315,3396,80946637,escort,,5,WALK,-0.3645610337754769 +647573013,1974307,760593,work,1,False,2,3347,3316,80946626,work,14.620790396011605,16,WALK,0.9232328252322722 +647573014,1974307,760593,work,2,False,2,3396,3347,80946626,home,,17,DRIVEALONEFREE,0.22369528704927913 +647573097,1974308,760593,escort,1,True,1,3315,3396,80946637,escort,,5,WALK,-0.36456103377547694 647573101,1974308,760593,escort,1,False,1,3396,3315,80946637,home,,5,WALK,-0.364561268866216 648387521,1976791,761445,escort,1,True,1,3314,3281,81048440,escort,,13,DRIVEALONEFREE,0.5167039708675432 648387525,1976791,761445,escort,1,False,1,3281,3314,81048440,home,,13,SHARED2FREE,0.5107954158143181 648387809,1976792,761445,atwork,1,True,1,3176,3176,81048476,atwork,,10,WALK,2.6706195748553934 648387813,1976792,761445,atwork,1,False,1,3176,3176,81048476,work,,10,WALK,2.6706195748553934 648387977,1976792,761445,othdiscr,1,True,1,3398,3281,81048497,othdiscr,,14,DRIVEALONEFREE,0.3504071806347388 -648387981,1976792,761445,othdiscr,1,False,1,3281,3398,81048497,home,,16,DRIVEALONEFREE,0.4048446305119533 +648387981,1976792,761445,othdiscr,1,False,1,3281,3398,81048497,home,,16,DRIVEALONEFREE,0.40484463051195335 648388089,1976792,761445,work,1,True,1,3176,3281,81048511,work,,5,WALK,-0.5120632445711146 648388093,1976792,761445,work,1,False,1,3281,3176,81048511,home,,14,DRIVEALONEFREE,-0.641534040627387 649042753,1978788,762159,social,1,True,1,3498,3598,81130344,social,,8,SHARED3FREE,0.7936471422825703 @@ -142,11 +142,11 @@ trip_id,person_id,household_id,primary_purpose,trip_num,outbound,trip_count,dest 649043193,1978790,762159,escort,1,True,1,3410,3598,81130399,escort,,8,SHARED2FREE,0.5634483478050719 649043197,1978790,762159,escort,1,False,1,3598,3410,81130399,home,,8,WALK,0.5562021020505422 649043433,1978790,762159,work,1,True,1,3574,3598,81130429,work,,8,DRIVEALONEFREE,-0.3302917776537384 -649043437,1978790,762159,work,1,False,1,3598,3574,81130429,home,,17,DRIVEALONEFREE,-0.3281436288428634 +649043437,1978790,762159,work,1,False,1,3598,3574,81130429,home,,17,DRIVEALONEFREE,-0.32814362884286347 649043761,1978791,762159,work,1,True,1,3176,3598,81130470,work,,7,DRIVEALONEFREE,-0.0843772599901702 649043765,1978791,762159,work,1,False,1,3598,3176,81130470,home,,17,SHARED2FREE,-0.3190406065799127 -819359665,2498047,922602,school,1,True,1,3340,3383,102419958,school,,9,WALK,0.0009173412871742 -819359669,2498047,922602,school,1,False,1,3383,3340,102419958,home,,15,WALK_LRF,0.0131918923923682 +819359665,2498047,922602,school,1,True,1,3340,3383,102419958,school,,9,WALK,0.0009173412871749596 +819359669,2498047,922602,school,1,False,1,3383,3340,102419958,home,,15,WALK_LRF,0.013191892392368922 819360057,2498048,922602,work,1,True,1,3383,3383,102420007,work,,7,WALK,0.6883532259078515 819360061,2498048,922602,work,1,False,1,3383,3383,102420007,home,,16,WALK,0.688353222623592 819360385,2498049,922602,work,1,True,1,3115,3383,102420048,work,,6,BIKE,-1.6833752824565824 @@ -155,26 +155,26 @@ trip_id,person_id,household_id,primary_purpose,trip_num,outbound,trip_count,dest 860079229,2622192,952720,school,1,False,1,3505,3378,107509903,home,,14,WALK,-0.3024751617974213 860079377,2622193,952720,escort,1,True,1,3376,3505,107509922,escort,,6,DRIVEALONEFREE,0.5385189014110604 860079381,2622193,952720,escort,1,False,1,3505,3376,107509922,home,,6,SHARED2FREE,0.5341382626062584 -860079529,2622193,952720,othmaint,1,True,1,3176,3505,107509941,othmaint,,8,DRIVEALONEFREE,0.4138401650211165 -860079533,2622193,952720,othmaint,1,False,1,3505,3176,107509941,home,,13,TAXI,0.4250830718473121 -860079897,2622194,952720,shopping,1,True,1,3131,3505,107509987,shopping,,9,TNC_SINGLE,0.0622325149119334 -860079901,2622194,952720,shopping,1,False,1,3505,3131,107509987,home,,14,TNC_SINGLE,-0.0694219397215716 -860080273,2622195,952720,work,1,True,1,3363,3505,107510034,work,,8,DRIVEALONEFREE,0.1605413267290312 -860080277,2622195,952720,work,1,False,1,3505,3363,107510034,home,,17,DRIVEALONEFREE,0.0796511780693144 -933123249,2844887,1028031,work,1,True,1,3497,3597,116640406,work,,11,DRIVEALONEFREE,-0.0640441703152012 +860079529,2622193,952720,othmaint,1,True,1,3176,3505,107509941,othmaint,,8,DRIVEALONEFREE,0.41384016502111654 +860079533,2622193,952720,othmaint,1,False,1,3505,3176,107509941,home,,13,TAXI,0.42508307184731214 +860079897,2622194,952720,shopping,1,True,1,3131,3505,107509987,shopping,,9,TNC_SINGLE,0.06223251491193346 +860079901,2622194,952720,shopping,1,False,1,3505,3131,107509987,home,,14,TNC_SINGLE,-0.06942193972157161 +860080273,2622195,952720,work,1,True,1,3363,3505,107510034,work,,8,DRIVEALONEFREE,0.16054132672903124 +860080277,2622195,952720,work,1,False,1,3505,3363,107510034,home,,17,DRIVEALONEFREE,0.07965117806931443 +933123249,2844887,1028031,work,1,True,1,3497,3597,116640406,work,,11,DRIVEALONEFREE,-0.06404417031520128 933123253,2844887,1028031,work,1,False,2,3410,3497,116640406,eatout,11.389365156532708,18,DRIVEALONEFREE,-0.2404374327805191 -933123254,2844887,1028031,work,2,False,2,3597,3410,116640406,home,,19,DRIVEALONEFREE,-0.1519509037830223 -962301409,2933845,1048898,school,1,True,1,3536,3134,120287676,school,,12,SHARED3FREE,-1.8253639025348931 -962301413,2933845,1048898,school,1,False,1,3134,3536,120287676,home,,22,SHARED3FREE,-1.800035982127559 +933123254,2844887,1028031,work,2,False,2,3597,3410,116640406,home,,19,DRIVEALONEFREE,-0.15195090378302237 +962301409,2933845,1048898,school,1,True,1,3536,3134,120287676,school,,12,SHARED3FREE,-1.8253639025348933 +962301413,2933845,1048898,school,1,False,1,3134,3536,120287676,home,,22,SHARED3FREE,-1.8000359821275593 962301737,2933846,1048898,school,1,True,1,3536,3134,120287717,school,,8,WALK,-2.1374786293216195 962301741,2933846,1048898,school,1,False,1,3134,3536,120287717,home,,16,SHARED2FREE,-2.1233856637960864 -962302017,2933847,1048898,othdiscr,1,True,1,3272,3134,120287752,othdiscr,,7,WALK,0.9723049345876644 +962302017,2933847,1048898,othdiscr,1,True,1,3272,3134,120287752,othdiscr,,7,WALK,0.9723049345876645 962302021,2933847,1048898,othdiscr,1,False,2,3399,3272,120287752,eatout,14.6590717484754,12,WALK,1.2236873568553286 962302022,2933847,1048898,othdiscr,2,False,2,3134,3399,120287752,home,,12,DRIVEALONEFREE,0.8157972936115587 962302457,2933848,1048898,work,1,True,1,3536,3134,120287807,work,,6,DRIVEALONEFREE,-0.1804673034758373 -962302461,2933848,1048898,work,1,False,2,3460,3536,120287807,escort,12.158136904433114,17,DRIVEALONEFREE,-0.2382160454714425 +962302461,2933848,1048898,work,1,False,2,3460,3536,120287807,escort,12.158136904433114,17,DRIVEALONEFREE,-0.23821604547144257 962302462,2933848,1048898,work,2,False,2,3134,3460,120287807,home,,18,DRIVEALONEFREE,-0.3950033708360319 1055052265,3216622,1148260,univ,1,True,1,3112,3098,131881533,univ,,14,WALK,-1.7221539064441829 -1055052269,3216622,1148260,univ,1,False,3,3112,3112,131881533,univ,9.085902358698815,16,WALK,-0.5510518904620926 -1055052270,3216622,1148260,univ,2,False,3,3176,3112,131881533,escort,14.025105605160244,17,WALK,-0.6958229859822935 +1055052269,3216622,1148260,univ,1,False,3,3112,3112,131881533,univ,9.085902358698817,16,WALK,-0.5510518904620926 +1055052270,3216622,1148260,univ,2,False,3,3176,3112,131881533,escort,14.025105605160245,17,WALK,-0.6958229859822935 1055052271,3216622,1148260,univ,3,False,3,3098,3176,131881533,home,,17,WALK,2.3965343775069243 diff --git a/activitysim/examples/placeholder_sandag/test/regress/final_1_zone_trips_sh.csv b/activitysim/examples/placeholder_sandag/test/regress/final_1_zone_trips_sh.csv new file mode 100644 index 0000000000..3e4eecce69 --- /dev/null +++ b/activitysim/examples/placeholder_sandag/test/regress/final_1_zone_trips_sh.csv @@ -0,0 +1,180 @@ +trip_id,person_id,household_id,primary_purpose,trip_num,outbound,trip_count,destination,origin,tour_id,purpose,destination_logsum,depart,trip_mode,mode_choice_logsum +10872201,33146,12593,work,1,True,3,3290,3259,1359025,escort,12.505695878319539,5,WALK,0.8796640848644649 +10872202,33146,12593,work,2,True,3,3317,3290,1359025,escort,12.74437072347788,7,WALK,0.7528069371143918 +10872203,33146,12593,work,3,True,3,3131,3317,1359025,work,,7,SHARED2FREE,-0.27715849105678125 +10872205,33146,12593,work,1,False,1,3259,3131,1359025,home,,16,SHARED2FREE,-0.3682899649069351 +10872529,33147,12593,work,1,True,1,3342,3259,1359066,work,,8,SHARED2FREE,0.4187992443742946 +10872533,33147,12593,work,1,False,4,3383,3342,1359066,escort,14.576775128994838,12,SHARED2FREE,0.8686143194682551 +10872534,33147,12593,work,2,False,4,3274,3383,1359066,social,13.66341661662671,13,SHARED2FREE,0.5107943560451084 +10872535,33147,12593,work,3,False,4,3340,3274,1359066,escort,14.112546915053883,13,SHARED2FREE,0.4230067105850933 +10872536,33147,12593,work,4,False,4,3259,3340,1359066,home,,13,DRIVEALONEFREE,0.24911633540131922 +11957177,36454,13797,shopping,1,True,1,3342,3134,1494647,shopping,,14,DRIVEALONEFREE,0.29416859071931833 +11957181,36454,13797,shopping,1,False,2,3574,3342,1494647,shopping,12.641799100811618,14,TAXI,-0.10899026335766256 +11957182,36454,13797,shopping,2,False,2,3134,3574,1494647,home,,14,TNC_SINGLE,-0.18417644456671423 +11957185,36454,13797,shopping,1,True,2,3402,3134,1494648,escort,17.45390698273753,16,DRIVEALONEFREE,1.2790849131748439 +11957186,36454,13797,shopping,2,True,2,3497,3402,1494648,shopping,,16,SHARED2FREE,1.520377820514991 +11957189,36454,13797,shopping,1,False,1,3134,3497,1494648,home,,16,SHARED2FREE,1.253344921201897 +11957553,36455,13797,work,1,True,1,3317,3134,1494694,work,,7,WALK,0.5467805012552107 +11957557,36455,13797,work,1,False,1,3134,3317,1494694,home,,23,WALK,0.533638185034308 +13679289,41705,15777,eatout,1,True,1,3437,3342,1709911,eatout,,10,DRIVEALONEFREE,1.1523919906663322 +13679293,41705,15777,eatout,1,False,1,3342,3437,1709911,home,,10,SHARED2FREE,1.125349223784251 +13679569,41706,15777,atwork,1,True,1,3131,3398,1709946,atwork,,12,SHARED2FREE,1.606572085549667 +13679573,41706,15777,atwork,1,False,1,3398,3131,1709946,work,,17,SHARED2FREE,1.565177581330629 +13679881,41706,15777,work,1,True,1,3398,3342,1709985,work,,8,DRIVEALONEFREE,0.229309200564582 +13679885,41706,15777,work,1,False,2,3317,3398,1709985,escort,12.99906661478402,18,DRIVEALONEFREE,0.3635288368935579 +13679886,41706,15777,work,2,False,2,3342,3317,1709985,home,,19,WALK,0.19920388273033734 +16411649,50035,18261,social,1,True,1,3497,3493,2051456,social,,15,SHARED2FREE,0.8144179602536189 +16411653,50035,18261,social,1,False,3,3398,3497,2051456,social,13.607997193750748,16,SHARED2FREE,0.9055182726514673 +16411654,50035,18261,social,2,False,3,3259,3398,2051456,social,14.227902019942242,16,SHARED2FREE,0.48164035102557873 +16411655,50035,18261,social,3,False,3,3493,3259,2051456,home,,16,TAXI,0.4624895573096613 +16411729,50035,18261,school,1,True,1,3340,3493,2051466,school,,7,SHARED3FREE,-0.3581378319265033 +16411733,50035,18261,school,1,False,1,3493,3340,2051466,home,,12,DRIVEALONEFREE,-0.7018451340834048 +16411745,50035,18261,shopping,1,True,1,3396,3493,2051468,shopping,,18,DRIVEALONEFREE,0.2675492794224116 +16411749,50035,18261,shopping,1,False,1,3493,3396,2051468,home,,21,DRIVEALONEFREE,0.26411413980416254 +16412073,50036,18261,shopping,1,True,1,3497,3493,2051509,shopping,,11,WALK,0.3815001484705561 +16412077,50036,18261,shopping,1,False,2,3378,3497,2051509,escort,12.868283363745013,11,DRIVEALONEFREE,-0.16091822502410236 +16412078,50036,18261,shopping,2,False,2,3493,3378,2051509,home,,11,DRIVEALONEFREE,-0.2842450511223511 +16412449,50037,18261,work,1,True,1,3396,3493,2051556,work,,7,DRIVEALONEFREE,0.2543385882925606 +16412453,50037,18261,work,1,False,1,3493,3396,2051556,home,,14,DRIVEALONEFREE,0.2544109368585356 +18151113,55338,19758,school,1,True,1,3592,3592,2268889,school,,7,WALK,-0.5037709043643227 +18151117,55338,19758,school,1,False,1,3592,3592,2268889,home,,10,WALK,-0.5037709043643227 +18151505,55339,19758,work,1,True,1,3497,3592,2268938,work,,5,WALK,-0.03916734678520527 +18151509,55339,19758,work,1,False,1,3592,3497,2268938,home,,16,WALK,-0.039161801335005776 +18990529,57897,20552,work,1,True,1,3176,3746,2373816,work,,7,DRIVEALONEFREE,-0.8320971690146864 +18990533,57897,20552,work,1,False,1,3746,3176,2373816,home,,20,DRIVEALONEFREE,-1.0379874213845641 +18990577,57898,20552,atwork,1,True,1,3410,3347,2373822,atwork,,13,DRIVEALONEFREE,0.15721645183778177 +18990581,57898,20552,atwork,1,False,1,3347,3410,2373822,work,,14,DRIVEALONEFREE,0.15656163392285075 +18990857,57898,20552,work,1,True,1,3347,3746,2373857,work,,11,DRIVEALONEFREE,-0.2916992537280374 +18990861,57898,20552,work,1,False,2,3849,3347,2373857,escort,11.83577521389007,15,DRIVEALONEFREE,-0.2521012321385745 +18990862,57898,20552,work,2,False,2,3746,3849,2373857,home,,15,DRIVEALONEFREE,-0.0911874930860481 +18991185,57899,20552,work,1,True,1,3402,3746,2373898,work,,7,WALK,-1.1548338964177454 +18991189,57899,20552,work,1,False,1,3746,3402,2373898,home,,17,WALK,-1.229178185082696 +18991841,57901,20552,work,1,True,1,3115,3746,2373980,work,,6,DRIVEALONEFREE,0.19574077710207732 +18991845,57901,20552,work,1,False,1,3746,3115,2373980,home,,12,DRIVEALONEFREE,0.20779404762909298 +18991849,57901,20552,work,1,True,2,3460,3746,2373981,othmaint,12.489995670716882,15,SHARED2FREE,0.24946092202202907 +18991850,57901,20552,work,2,True,2,3115,3460,2373981,work,,16,DRIVEALONEFREE,0.10597046751418379 +18991853,57901,20552,work,1,False,1,3746,3115,2373981,home,,20,SHARED2FREE,0.23660752783217825 +20510417,62531,21869,school,1,True,1,3460,3316,2563802,school,,20,SHARED3FREE,-1.4448137456466916 +20510421,62531,21869,school,1,False,1,3316,3460,2563802,home,,20,WALK,-1.5207459403958272 +20510569,62532,21869,escort,1,True,1,3398,3316,2563821,escort,,6,SHARED2FREE,0.17869598454022895 +20510573,62532,21869,escort,1,False,1,3316,3398,2563821,home,,7,DRIVEALONEFREE,0.20045149458253975 +20510897,62533,21869,escort,1,True,1,3402,3316,2563862,escort,,5,SHARED3FREE,0.7112775892674524 +20510901,62533,21869,escort,1,False,4,3339,3402,2563862,shopping,14.091505819104302,6,SHARED3FREE,0.983638157033812 +20510902,62533,21869,escort,2,False,4,3505,3339,2563862,eatout,15.083442995634531,6,SHARED3FREE,0.9900387210738842 +20510903,62533,21869,escort,3,False,4,3536,3505,2563862,escort,15.770594264076767,6,SHARED2FREE,0.7686807718987421 +20510904,62533,21869,escort,4,False,4,3316,3536,2563862,home,,6,SHARED3FREE,0.5576305299679216 +20510905,62533,21869,escort,1,True,1,3519,3316,2563863,escort,,11,SHARED2FREE,0.4802732339996155 +20510909,62533,21869,escort,1,False,1,3316,3519,2563863,home,,11,SHARED2FREE,0.48172719450388046 +20510913,62533,21869,escort,1,True,1,3398,3316,2563864,escort,,14,SHARED2FREE,0.5739525286758124 +20510917,62533,21869,escort,1,False,1,3316,3398,2563864,home,,14,DRIVEALONEFREE,0.5863871055816903 +20511025,62533,21869,othdiscr,1,True,1,3383,3316,2563878,othdiscr,,11,DRIVEALONEFREE,-0.03084494420474515 +20511029,62533,21869,othdiscr,1,False,1,3316,3383,2563878,home,,11,DRIVEALONEFREE,-0.041632432068551585 +20511401,62534,21869,school,1,True,1,3316,3316,2563925,school,,8,WALK,-0.7509920520073805 +20511405,62534,21869,school,1,False,2,3363,3316,2563925,shopping,5.114003115539301,9,SHARED3FREE,-1.3212813014651676 +20511406,62534,21869,school,2,False,2,3316,3363,2563925,home,,9,WALK,-1.2832199854288413 +22303745,67999,23619,escort,1,True,1,3410,3378,2787968,escort,,13,TNC_SHARED,0.23700032543064115 +22303749,67999,23619,escort,1,False,3,3822,3410,2787968,eatout,12.793892361239553,13,TNC_SINGLE,-0.021948901498790654 +22303750,67999,23619,escort,2,False,3,3412,3822,2787968,escort,12.238309717426683,13,TNC_SINGLE,-0.1578164617938098 +22303751,67999,23619,escort,3,False,3,3378,3412,2787968,home,,13,TNC_SINGLE,0.11060084957709816 +22303961,67999,23619,social,1,True,1,3400,3378,2787995,social,,17,WALK,0.02192713181801061 +22303965,67999,23619,social,1,False,1,3378,3400,2787995,home,,20,WALK,0.021942143743783993 +22304313,68000,23619,work,1,True,1,3375,3378,2788039,work,,7,WALK,0.16348447779787817 +22304317,68000,23619,work,1,False,1,3378,3375,2788039,home,,20,WALK,0.16348447779787817 +25904705,78977,26897,school,1,True,1,3339,3339,3238088,school,,7,WALK,0.3224231573284306 +25904709,78977,26897,school,1,False,1,3339,3339,3238088,home,,14,WALK,0.3224234088552417 +421021769,1283602,435012,work,1,True,1,3410,3315,52627721,work,,8,DRIVEALONEFREE,-0.7004813704631009 +421021773,1283602,435012,work,1,False,2,3314,3410,52627721,social,10.399929897293942,14,DRIVEALONEFREE,-0.7505287871501307 +421021774,1283602,435012,work,2,False,2,3315,3314,52627721,home,,18,DRIVEALONEFREE,0.1928245919289156 +421108753,1283868,435278,eatout,1,True,1,3314,3314,52638594,eatout,,18,WALK,0.24082656084284254 +421108757,1283868,435278,eatout,1,False,1,3314,3314,52638594,home,,21,WALK,0.24082693725080942 +421108889,1283868,435278,atwork,1,True,1,3176,3176,52638611,atwork,,16,WALK,4.4100666659595396 +421108893,1283868,435278,atwork,1,False,1,3176,3176,52638611,work,,16,WALK,4.4100666659595396 +421109017,1283868,435278,work,1,True,1,3176,3314,52638627,work,,9,DRIVEALONEFREE,-0.8523980347217911 +421109021,1283868,435278,work,1,False,2,3176,3176,52638627,work,14.061231380263717,18,WALK,2.2993936628468057 +421109022,1283868,435278,work,2,False,2,3314,3176,52638627,home,,18,DRIVEALONEFREE,-0.8993747376742869 +421134601,1283946,435356,work,1,True,1,3619,3315,52641825,work,,7,TNC_SINGLE,0.3084730568128902 +421134605,1283946,435356,work,1,False,1,3315,3619,52641825,home,,18,TNC_SINGLE,0.16884967311531862 +421348457,1284598,436008,work,1,True,1,3464,3261,52668557,work,,9,DRIVEALONEFREE,-0.19905580141140403 +421348461,1284598,436008,work,1,False,1,3261,3464,52668557,home,,18,DRIVEALONEFREE,-0.18990025215791376 +421878553,1286215,437625,atwork,1,True,1,3176,3176,52734819,atwork,,10,WALK,4.410066095224354 +421878557,1286215,437625,atwork,1,False,1,3176,3176,52734819,work,,13,WALK,4.410066095224354 +421878833,1286215,437625,work,1,True,1,3176,3317,52734854,work,,8,DRIVEALONEFREE,-0.7732401280141026 +421878837,1286215,437625,work,1,False,3,3398,3176,52734854,othmaint,13.419982162114987,17,DRIVEALONEFREE,-1.237602447505927 +421878838,1286215,437625,work,2,False,3,3397,3398,52734854,othmaint,12.007055480943817,19,DRIVEALONEFREE,0.32830019891532175 +421878839,1286215,437625,work,3,False,3,3317,3397,52734854,home,,19,WALK,0.19244201397140973 +423180617,1290184,441594,shopping,1,True,1,3131,3195,52897577,shopping,,9,SHARED2FREE,0.2504977565051515 +423180621,1290184,441594,shopping,1,False,1,3195,3131,52897577,home,,10,SHARED2FREE,0.23242459998329676 +423180625,1290184,441594,shopping,1,True,1,3131,3195,52897578,shopping,,16,WALK,-1.7981189655442733 +423180629,1290184,441594,shopping,1,False,1,3195,3131,52897578,home,,16,WALK,-1.891099527786877 +423180665,1290184,441594,work,1,True,1,3176,3195,52897583,work,,10,WALK,0.855151926514014 +423180669,1290184,441594,work,1,False,1,3195,3176,52897583,home,,16,WALK,0.9414955852678495 +423325361,1290626,442036,atwork,1,True,1,3340,3502,52915670,atwork,,10,BIKE,0.10458674181820576 +423325365,1290626,442036,atwork,1,False,2,3176,3340,52915670,othmaint,16.13095576158003,11,BIKE,-0.524564641781727 +423325366,1290626,442036,atwork,2,False,2,3502,3176,52915670,work,,11,BIKE,3.2768259555804935 +423325641,1290626,442036,work,1,True,1,3502,3278,52915705,work,,8,BIKE,-1.0096302214124557 +423325645,1290626,442036,work,1,False,3,3176,3502,52915705,othdiscr,13.17876515323236,18,BIKE,-1.546730219362895 +423325646,1290626,442036,work,2,False,3,3176,3176,52915705,othmaint,21.761313694214984,18,WALK,3.1160978512473827 +423325647,1290626,442036,work,3,False,3,3278,3176,52915705,home,,18,BIKE,2.471064663969691 +611033041,1862905,721960,othdiscr,1,True,1,3460,3598,76379130,othdiscr,,15,WALK,-1.234581024206817 +611033045,1862905,721960,othdiscr,1,False,1,3598,3460,76379130,home,,21,WALK,-0.9300094952392008 +611033369,1862906,721960,othdiscr,1,True,1,3705,3598,76379171,othdiscr,,11,WALK,-0.8123168897915816 +611033373,1862906,721960,othdiscr,1,False,1,3598,3705,76379171,home,,16,WALK,-0.6995691553182372 +647572569,1974306,760593,othdiscr,1,True,1,3176,3396,80946571,othdiscr,,8,SHARED2FREE,-0.2717262829172214 +647572573,1974306,760593,othdiscr,1,False,1,3396,3176,80946571,home,,13,SHARED2FREE,-0.3489570743749156 +647572729,1974307,760593,atwork,1,True,1,3574,3316,80946591,atwork,,12,TNC_SINGLE,0.47036512616407533 +647572733,1974307,760593,atwork,1,False,1,3316,3574,80946591,work,,13,TNC_SINGLE,0.32774303107179875 +647573009,1974307,760593,work,1,True,2,3400,3396,80946626,shopping,12.693917677139323,9,SHARED2FREE,0.8051982291809634 +647573010,1974307,760593,work,2,True,2,3316,3400,80946626,work,,10,SHARED2FREE,0.3488831452765107 +647573013,1974307,760593,work,1,False,2,3347,3316,80946626,work,14.620790396011605,16,WALK,0.9232328252322722 +647573014,1974307,760593,work,2,False,2,3396,3347,80946626,home,,17,DRIVEALONEFREE,0.22369528704927913 +647573097,1974308,760593,escort,1,True,1,3315,3396,80946637,escort,,5,WALK,-0.36456103377547694 +647573101,1974308,760593,escort,1,False,1,3396,3315,80946637,home,,5,WALK,-0.364561268866216 +648387521,1976791,761445,escort,1,True,1,3314,3281,81048440,escort,,13,DRIVEALONEFREE,0.5167039708675432 +648387525,1976791,761445,escort,1,False,1,3281,3314,81048440,home,,13,SHARED2FREE,0.5107954158143181 +648387809,1976792,761445,atwork,1,True,1,3176,3176,81048476,atwork,,10,WALK,2.6706195748553934 +648387813,1976792,761445,atwork,1,False,1,3176,3176,81048476,work,,10,WALK,2.6706195748553934 +648387977,1976792,761445,othdiscr,1,True,1,3398,3281,81048497,othdiscr,,14,DRIVEALONEFREE,0.3504071806347388 +648387981,1976792,761445,othdiscr,1,False,1,3281,3398,81048497,home,,16,DRIVEALONEFREE,0.40484463051195335 +648388089,1976792,761445,work,1,True,1,3176,3281,81048511,work,,5,WALK,-0.5120632445711146 +648388093,1976792,761445,work,1,False,1,3281,3176,81048511,home,,14,DRIVEALONEFREE,-0.641534040627387 +649042753,1978788,762159,social,1,True,1,3498,3598,81130344,social,,8,SHARED3FREE,0.7936471422825703 +649042757,1978788,762159,social,1,False,1,3598,3498,81130344,home,,20,SHARED3FREE,0.8165066327406539 +649043193,1978790,762159,escort,1,True,1,3410,3598,81130399,escort,,8,SHARED2FREE,0.5634483478050719 +649043197,1978790,762159,escort,1,False,1,3598,3410,81130399,home,,8,WALK,0.5562021020505422 +649043433,1978790,762159,work,1,True,1,3574,3598,81130429,work,,8,DRIVEALONEFREE,-0.3302917776537384 +649043437,1978790,762159,work,1,False,1,3598,3574,81130429,home,,17,DRIVEALONEFREE,-0.32814362884286347 +649043761,1978791,762159,work,1,True,1,3176,3598,81130470,work,,7,DRIVEALONEFREE,-0.0843772599901702 +649043765,1978791,762159,work,1,False,1,3598,3176,81130470,home,,17,SHARED2FREE,-0.3190406065799127 +819359665,2498047,922602,school,1,True,1,3340,3383,102419958,school,,9,WALK,0.0009173412871749596 +819359669,2498047,922602,school,1,False,1,3383,3340,102419958,home,,15,WALK_LRF,0.013191892392368922 +819360057,2498048,922602,work,1,True,1,3383,3383,102420007,work,,7,WALK,0.6883532259078515 +819360061,2498048,922602,work,1,False,1,3383,3383,102420007,home,,16,WALK,0.688353222623592 +819360385,2498049,922602,work,1,True,1,3115,3383,102420048,work,,6,BIKE,-1.6833752824565824 +819360389,2498049,922602,work,1,False,1,3383,3115,102420048,home,,16,BIKE,-1.406426614368767 +860079225,2622192,952720,school,1,True,1,3378,3505,107509903,school,,7,WALK,-0.3024781465081771 +860079229,2622192,952720,school,1,False,1,3505,3378,107509903,home,,14,WALK,-0.3024751617974213 +860079377,2622193,952720,escort,1,True,1,3376,3505,107509922,escort,,6,DRIVEALONEFREE,0.5385189014110604 +860079381,2622193,952720,escort,1,False,1,3505,3376,107509922,home,,6,SHARED2FREE,0.5341382626062584 +860079529,2622193,952720,othmaint,1,True,1,3176,3505,107509941,othmaint,,8,DRIVEALONEFREE,0.41384016502111654 +860079533,2622193,952720,othmaint,1,False,1,3505,3176,107509941,home,,13,TAXI,0.42508307184731214 +860079897,2622194,952720,shopping,1,True,1,3131,3505,107509987,shopping,,9,TNC_SINGLE,0.06223251491193346 +860079901,2622194,952720,shopping,1,False,1,3505,3131,107509987,home,,14,TNC_SINGLE,-0.06942193972157161 +860080273,2622195,952720,work,1,True,1,3363,3505,107510034,work,,8,DRIVEALONEFREE,0.16054132672903124 +860080277,2622195,952720,work,1,False,1,3505,3363,107510034,home,,17,DRIVEALONEFREE,0.07965117806931443 +933123249,2844887,1028031,work,1,True,1,3497,3597,116640406,work,,11,DRIVEALONEFREE,-0.06404417031520128 +933123253,2844887,1028031,work,1,False,2,3410,3497,116640406,eatout,11.389365156532708,18,DRIVEALONEFREE,-0.2404374327805191 +933123254,2844887,1028031,work,2,False,2,3597,3410,116640406,home,,19,DRIVEALONEFREE,-0.15195090378302237 +962301409,2933845,1048898,school,1,True,1,3536,3134,120287676,school,,12,SHARED3FREE,-1.8253639025348933 +962301413,2933845,1048898,school,1,False,1,3134,3536,120287676,home,,22,SHARED3FREE,-1.8000359821275593 +962301737,2933846,1048898,school,1,True,1,3536,3134,120287717,school,,8,WALK,-2.1374786293216195 +962301741,2933846,1048898,school,1,False,1,3134,3536,120287717,home,,16,SHARED2FREE,-2.1233856637960864 +962302017,2933847,1048898,othdiscr,1,True,1,3272,3134,120287752,othdiscr,,7,WALK,0.9723049345876645 +962302021,2933847,1048898,othdiscr,1,False,2,3399,3272,120287752,eatout,14.6590717484754,12,WALK,1.2236873568553286 +962302022,2933847,1048898,othdiscr,2,False,2,3134,3399,120287752,home,,12,DRIVEALONEFREE,0.8157972936115587 +962302457,2933848,1048898,work,1,True,1,3536,3134,120287807,work,,6,DRIVEALONEFREE,-0.1804673034758373 +962302461,2933848,1048898,work,1,False,2,3460,3536,120287807,escort,12.158136904433114,17,DRIVEALONEFREE,-0.23821604547144257 +962302462,2933848,1048898,work,2,False,2,3134,3460,120287807,home,,18,DRIVEALONEFREE,-0.3950033708360319 +1055052265,3216622,1148260,univ,1,True,1,3112,3098,131881533,univ,,14,WALK,-1.7221539064441829 +1055052269,3216622,1148260,univ,1,False,3,3112,3112,131881533,univ,9.085902358698817,16,WALK,-0.5510518904620926 +1055052270,3216622,1148260,univ,2,False,3,3176,3112,131881533,escort,14.025105605160245,17,WALK,-0.6958229859822935 +1055052271,3216622,1148260,univ,3,False,3,3098,3176,131881533,home,,17,WALK,2.3965343775069243 diff --git a/activitysim/examples/placeholder_sandag/test/regress/final_2_zone_tours_sh.csv b/activitysim/examples/placeholder_sandag/test/regress/final_2_zone_tours_sh.csv new file mode 100644 index 0000000000..cdf5efb675 --- /dev/null +++ b/activitysim/examples/placeholder_sandag/test/regress/final_2_zone_tours_sh.csv @@ -0,0 +1,81 @@ +tour_id,person_id,tour_type,tour_type_count,tour_type_num,tour_num,tour_count,tour_category,number_of_participants,destination,origin,household_id,tdd,start,end,duration,composition,destination_logsum,tour_mode,mode_choice_logsum,atwork_subtour_frequency,parent_tour_id,stop_frequency,primary_purpose +1359025,33146,work,1,1,1,1,mandatory,1,1100,560,12593,13.0,5.0,18.0,13.0,,,SHARED2FREE,0.36538018644207043,no_subtours,,2out_0in,work +1359066,33147,work,1,1,1,1,mandatory,1,1070,560,12593,59.0,8.0,13.0,5.0,,,SHARED3FREE,0.09941178487064588,no_subtours,,0out_3in,work +1494647,36454,shopping,2,1,1,2,non_mandatory,1,725,580,13797,128.0,13.0,17.0,4.0,,11.588883,SHARED2FREE,-0.0992827123431451,,,0out_1in,shopping +1494648,36454,shopping,2,2,2,2,non_mandatory,1,729,580,13797,169.0,18.0,18.0,0.0,,11.589278,WALK,0.22215181107342957,,,0out_2in,shopping +1494680,36455,othdiscr,1,1,1,1,non_mandatory,1,621,580,13797,159.0,16.0,21.0,5.0,,12.871272,DRIVEALONEFREE,0.3181692351526671,,,0out_2in,othdiscr +1494694,36455,work,1,1,1,1,mandatory,1,555,580,13797,58.0,8.0,12.0,4.0,,,WALK,0.4370819786779232,no_subtours,,0out_0in,work +1709911,41705,eatout,1,1,1,1,non_mandatory,1,1070,645,15777,76.0,9.0,15.0,6.0,,11.872931,SHARED2FREE,-0.7287327063424008,,,0out_0in,eatout +1709946,41706,business,1,1,1,1,atwork,1,988,763,15777,116.0,12.0,16.0,4.0,,18.66777,WALK,0.137119353213824,,1709985.0,0out_0in,atwork +1709985,41706,work,1,1,1,1,mandatory,1,763,645,15777,65.0,8.0,19.0,11.0,,,DRIVEALONEFREE,0.441411411669672,business1,,0out_1in,work +2051448,50035,eatout,1,1,1,1,joint,3,986,757,18261,156.0,16.0,18.0,2.0,mixed,11.548074,BIKE,-4.315590206938891,,,0out_1in,eatout +2051466,50035,school,1,1,1,1,mandatory,1,919,757,18261,43.0,7.0,13.0,6.0,,,SHARED3FREE,-1.4990946171231887,,,0out_0in,school +2051468,50035,shopping,1,1,1,1,non_mandatory,1,991,757,18261,175.0,19.0,19.0,0.0,,11.432129,DRIVEALONEFREE,-0.49743352602371993,,,0out_0in,shopping +2051504,50036,othmaint,1,1,1,1,non_mandatory,1,874,757,18261,44.0,7.0,14.0,7.0,,12.126692,DRIVEALONEFREE,-0.20958674876271485,,,0out_0in,othmaint +2051556,50037,work,1,1,1,1,mandatory,1,1070,757,18261,46.0,7.0,16.0,9.0,,,DRIVEALONEFREE,-1.259416062462369,no_subtours,,0out_0in,work +2268889,55338,school,1,1,1,1,mandatory,1,684,794,19758,40.0,7.0,10.0,3.0,,,WALK,-1.256172025738767,,,0out_0in,school +2268938,55339,work,1,1,1,1,mandatory,1,699,794,19758,11.0,5.0,16.0,11.0,,,WALK,0.16257796649302067,no_subtours,,0out_0in,work +2373816,57897,work,1,1,1,1,mandatory,1,1070,829,20552,50.0,7.0,20.0,13.0,,,DRIVEALONEFREE,-0.30338142975106963,no_subtours,,0out_0in,work +2373822,57898,eat,1,1,1,1,atwork,1,1070,904,20552,125.0,13.0,14.0,1.0,,11.039771,DRIVEALONEFREE,-0.5383114957971221,,2373857.0,0out_0in,atwork +2373857,57898,work,1,1,1,1,mandatory,1,904,829,20552,103.0,11.0,15.0,4.0,,,DRIVEALONEFREE,0.41082359419482434,eat,,0out_1in,work +2373898,57899,work,1,1,1,1,mandatory,1,687,829,20552,47.0,7.0,17.0,10.0,,,WALK,1.295796532158761,no_subtours,,0out_0in,work +2373942,57901,business,1,1,1,1,atwork,1,990,706,20552,162.0,17.0,17.0,0.0,,11.247394,DRIVEALONEFREE,0.6159985419045241,,2373981.0,0out_0in,atwork +2373980,57901,work,2,1,1,2,mandatory,1,706,829,20552,25.0,6.0,12.0,6.0,,,SHARED2FREE,0.5358425735133627,no_subtours,,0out_0in,work +2373981,57901,work,2,2,2,2,mandatory,1,706,829,20552,151.0,15.0,21.0,6.0,,,DRIVEALONEFREE,0.5358031993849557,business1,,1out_0in,work +2563802,62531,school,1,1,1,1,mandatory,1,943,900,21869,181.0,20.0,21.0,1.0,,,SHARED3FREE,-1.226764637052788,,,0out_0in,school +2563821,62532,escort,1,1,1,1,non_mandatory,1,647,900,21869,20.0,6.0,7.0,1.0,,12.495263,SHARED2FREE,-1.2864424041376865,,,0out_0in,escort +2563862,62533,escort,3,1,1,4,non_mandatory,1,695,900,21869,1.0,5.0,6.0,1.0,,12.5569935,SHARED3FREE,-1.2677737730011323,,,0out_3in,escort +2563863,62533,escort,3,2,2,4,non_mandatory,1,518,900,21869,99.0,11.0,11.0,0.0,,12.4978695,SHARED2FREE,-1.6806379182094977,,,0out_0in,escort +2563864,62533,escort,3,3,3,4,non_mandatory,1,844,900,21869,135.0,14.0,14.0,0.0,,12.529105,SHARED3FREE,-1.4300972332149253,,,0out_0in,escort +2563878,62533,othdiscr,1,1,4,4,non_mandatory,1,1070,900,21869,100.0,11.0,12.0,1.0,,12.891527,DRIVEALONEFREE,-0.1277668073387934,,,0out_0in,othdiscr +2563925,62534,school,1,1,1,1,mandatory,1,916,900,21869,55.0,8.0,9.0,1.0,,,SHARED2FREE,-0.036284049053444734,,,0out_1in,school +2787968,67999,escort,1,1,1,2,non_mandatory,1,767,973,23619,124.0,13.0,13.0,0.0,,12.967598,TNC_SINGLE,-0.7500927100255308,,,0out_2in,escort +2787995,67999,social,1,1,2,2,non_mandatory,1,913,973,23619,165.0,17.0,20.0,3.0,,12.107248,WALK,1.0240294566725847,,,0out_0in,social +2788039,68000,work,1,1,1,1,mandatory,1,1044,973,23619,49.0,7.0,19.0,12.0,,,SHARED2FREE,-0.007107189151378274,no_subtours,,1out_1in,work +3238088,78977,school,1,1,1,1,mandatory,1,984,1081,26897,44.0,7.0,14.0,7.0,,,WALK,0.12444784050821488,,,0out_0in,school +3238143,78979,eat,1,1,1,1,atwork,1,877,871,26897,72.0,9.0,11.0,2.0,,18.13034,WALK,0.5663422365804629,,3238178.0,0out_0in,atwork +3238178,78979,work,1,1,1,1,mandatory,1,871,1081,26897,48.0,7.0,18.0,11.0,,,DRIVEALONEFREE,-0.014910558388757844,eat,,0out_0in,work +52627721,1283602,work,1,1,1,1,mandatory,1,1078,521,435012,64.0,8.0,18.0,10.0,,,DRIVEALONEFREE,-1.1965658036522242,no_subtours,,0out_1in,work +52638594,1283868,eatout,1,1,1,1,non_mandatory,1,1070,537,435278,172.0,18.0,21.0,3.0,,11.528167,SHARED2FREE,-1.1519134997360956,,,1out_0in,eatout +52638611,1283868,maint,1,1,1,1,atwork,1,713,923,435278,154.0,16.0,16.0,0.0,,18.36072,DRIVEALONEFREE,-0.27103028035417925,,52638627.0,0out_0in,atwork +52638627,1283868,work,1,1,1,1,mandatory,1,923,537,435278,79.0,9.0,18.0,9.0,,,DRIVEALONEFREE,-0.43545363448393387,maint,,0out_1in,work +52641825,1283946,work,1,1,1,1,mandatory,1,1005,523,435356,48.0,7.0,18.0,11.0,,,TNC_SINGLE,-0.15324518765147138,no_subtours,,0out_0in,work +52668557,1284598,work,1,1,1,1,mandatory,1,551,562,436008,80.0,9.0,19.0,10.0,,,DRIVEALONEFREE,0.5908314031713293,no_subtours,,0out_0in,work +52734819,1286215,eat,1,1,1,1,atwork,1,1077,606,437625,88.0,10.0,13.0,3.0,,18.12966,DRIVEALONEFREE,-1.0786065781751473,,52734854.0,0out_0in,atwork +52734854,1286215,work,1,1,1,1,mandatory,1,606,656,437625,65.0,8.0,19.0,11.0,,,DRIVEALONEFREE,0.34770109667460397,eat,,0out_2in,work +52897544,1290184,business,1,1,1,1,atwork,1,763,952,441594,99.0,11.0,11.0,0.0,,17.608831,DRIVEALONEFREE,-0.2638632508540483,,52897583.0,0out_0in,atwork +52897550,1290184,eatout,1,1,4,4,non_mandatory,1,946,1096,441594,181.0,20.0,21.0,1.0,,11.792316,DRIVEALONEFREE,-0.27383046206140366,,,0out_1in,eatout +52897569,1290184,othdiscr,3,1,1,4,non_mandatory,1,684,1096,441594,19.0,6.0,6.0,0.0,,12.786691,DRIVEALONEFREE,0.009380535229135779,,,0out_0in,othdiscr +52897570,1290184,othdiscr,3,2,2,4,non_mandatory,1,762,1096,441594,185.0,21.0,22.0,1.0,,12.837735,DRIVEALONEFREE,-0.10046306805513741,,,0out_0in,othdiscr +52897571,1290184,othdiscr,3,3,3,4,non_mandatory,1,725,1096,441594,189.0,23.0,23.0,0.0,,12.81655,DRIVEALONEFREE,0.10969887272263075,,,1out_0in,othdiscr +52897583,1290184,work,1,1,1,1,mandatory,1,952,1096,441594,81.0,9.0,20.0,11.0,,,DRIVEALONEFREE,-0.27927776481025,business1,,0out_1in,work +52915670,1290626,eat,1,1,1,1,atwork,1,1070,1070,442036,86.0,10.0,11.0,1.0,,19.461575,WALK,0.5471153271447997,,52915705.0,0out_1in,atwork +52915705,1290626,work,1,1,1,1,mandatory,1,1070,1100,442036,64.0,8.0,18.0,10.0,,,WALK_LRF,-0.4785075925903294,eat,,1out_0in,work +76379130,1862905,othdiscr,1,1,1,1,non_mandatory,1,975,960,721960,151.0,15.0,21.0,6.0,,12.772852,SHARED3FREE,0.21983705470524506,,,0out_0in,othdiscr +76379171,1862906,othdiscr,1,1,1,1,non_mandatory,1,980,960,721960,104.0,11.0,16.0,5.0,,12.850391,SHARED2FREE,0.22336853092936326,,,0out_1in,othdiscr +80946571,1974306,othdiscr,1,1,1,1,non_mandatory,1,564,509,760593,62.0,8.0,16.0,8.0,,13.379667,SHARED3FREE,1.1645177740991923,,,0out_0in,othdiscr +80946591,1974307,eat,1,1,1,1,atwork,1,853,696,760593,113.0,12.0,13.0,1.0,,18.752243,TNC_SINGLE,0.28357719040363577,,80946626.0,0out_0in,atwork +80946626,1974307,work,1,1,1,1,mandatory,1,696,509,760593,79.0,9.0,18.0,9.0,,,WALK,0.32952249428941843,eat,,1out_0in,work +80946637,1974308,escort,1,1,1,1,non_mandatory,1,716,509,760593,0.0,5.0,5.0,0.0,,12.669005,SHARED3FREE,-0.8421136510342611,,,0out_0in,escort +81048440,1976791,escort,1,1,1,1,non_mandatory,1,908,613,761445,124.0,13.0,13.0,0.0,,12.820903,SHARED2FREE,-0.9700895291217932,,,0out_0in,escort +81048476,1976792,eat,1,1,1,1,atwork,1,517,517,761445,70.0,9.0,9.0,0.0,,17.917187,DRIVEALONEFREE,1.039507486293145,,81048511.0,0out_0in,atwork +81048508,1976792,social,1,1,1,1,non_mandatory,1,831,613,761445,140.0,14.0,19.0,5.0,,12.067439,WALK,0.4636430464295935,,,0out_0in,social +81048511,1976792,work,1,1,1,1,mandatory,1,517,613,761445,7.0,5.0,12.0,7.0,,,DRIVEALONEFREE,0.03163141281794101,eat,,0out_0in,work +81130344,1978788,social,1,1,1,1,non_mandatory,1,739,961,762159,66.0,8.0,20.0,12.0,,11.906203,SHARED3FREE,0.6662526170664789,,,0out_0in,social +81130399,1978790,escort,1,1,1,1,non_mandatory,1,992,961,762159,54.0,8.0,8.0,0.0,,12.577145,SHARED2FREE,-0.8831308815924643,,,0out_0in,escort +81130429,1978790,work,1,1,1,1,mandatory,1,681,961,762159,63.0,8.0,17.0,9.0,,,DRIVEALONEFREE,-0.016977745800912,no_subtours,,0out_0in,work +81130470,1978791,work,1,1,1,1,mandatory,1,1070,961,762159,47.0,7.0,17.0,10.0,,,SHARED2FREE,-0.9280034994526589,no_subtours,,0out_0in,work +102419958,2498047,school,1,1,1,1,mandatory,1,737,730,922602,77.0,9.0,16.0,7.0,,,WALK,2.6747421575246584,,,0out_0in,school +102420007,2498048,work,1,1,1,1,mandatory,1,722,730,922602,46.0,7.0,16.0,9.0,,,WALK,2.041191360849634,no_subtours,,0out_0in,work +102420048,2498049,work,1,1,1,1,mandatory,1,755,730,922602,29.0,6.0,16.0,10.0,,,WALK,1.1684000891097426,no_subtours,,0out_0in,work +107509903,2622192,school,1,1,1,1,mandatory,1,1004,1025,952720,44.0,7.0,14.0,7.0,,,WALK,0.28186425446408825,,,0out_0in,school +107509922,2622193,escort,1,1,1,2,non_mandatory,1,773,1025,952720,19.0,6.0,6.0,0.0,,12.838724,SHARED2FREE,-0.7024294415801873,,,0out_0in,escort +107509941,2622193,othmaint,1,1,2,2,non_mandatory,1,550,1025,952720,61.0,8.0,15.0,7.0,,12.457189,DRIVEALONEFREE,-0.0718658187188987,,,0out_0in,othmaint +107509987,2622194,shopping,1,1,1,1,non_mandatory,1,989,1025,952720,72.0,9.0,11.0,2.0,,11.894073,WALK,0.541554508705619,,,0out_0in,shopping +107510034,2622195,work,1,1,1,1,mandatory,1,1021,1025,952720,63.0,8.0,17.0,9.0,,,DRIVEALONEFREE,0.576165950576846,no_subtours,,0out_0in,work +116640406,2844887,work,1,1,1,1,mandatory,1,796,846,1028031,109.0,11.0,21.0,10.0,,,DRIVEALONEFREE,0.411732781392635,no_subtours,,0out_1in,work +120287676,2933845,school,1,1,1,1,mandatory,1,563,574,1048898,121.0,12.0,21.0,9.0,,,WALK,0.07010832880583706,,,0out_0in,school +120287717,2933846,school,1,1,1,1,mandatory,1,515,574,1048898,62.0,8.0,16.0,8.0,,,SHARED2FREE,-1.0988306831746049,,,0out_0in,school +120287752,2933847,othdiscr,1,1,1,1,non_mandatory,1,623,574,1048898,42.0,7.0,12.0,5.0,,12.852489,DRIVEALONEFREE,0.12407657987930228,,,0out_1in,othdiscr +120287807,2933848,work,1,1,1,1,mandatory,1,502,574,1048898,31.0,6.0,18.0,12.0,,,DRIVEALONEFREE,0.25163331437904624,no_subtours,,0out_1in,work +131881533,3216622,school,1,1,1,1,mandatory,1,1074,1076,1148260,136.0,14.0,15.0,1.0,,,WALK,-0.49868912988606207,,,0out_2in,univ diff --git a/activitysim/examples/placeholder_sandag/test/regress/final_2_zone_trips_sh.csv b/activitysim/examples/placeholder_sandag/test/regress/final_2_zone_trips_sh.csv new file mode 100644 index 0000000000..b4f546bd09 --- /dev/null +++ b/activitysim/examples/placeholder_sandag/test/regress/final_2_zone_trips_sh.csv @@ -0,0 +1,200 @@ +trip_id,person_id,household_id,primary_purpose,trip_num,outbound,trip_count,destination,origin,tour_id,purpose,destination_logsum,depart,trip_mode,mode_choice_logsum +10872201,33146,12593,work,1,True,3,919,560,1359025,escort,13.797406,5,SHARED2FREE,0.10106917792944203 +10872202,33146,12593,work,2,True,3,997,919,1359025,escort,14.823335,7,WALK,0.919706074693934 +10872203,33146,12593,work,3,True,3,1100,997,1359025,work,,7,WALK,0.828896351501718 +10872205,33146,12593,work,1,False,1,560,1100,1359025,home,,18,SHARED2FREE,0.1467435629753655 +10872529,33147,12593,work,1,True,1,1070,560,1359066,work,,8,SHARED3FREE,-0.4079095067864641 +10872533,33147,12593,work,1,False,4,1070,1070,1359066,escort,13.64208,12,WALK,1.909059923629201 +10872534,33147,12593,work,2,False,4,1070,1070,1359066,social,14.027128,13,WALK,1.916676460871733 +10872535,33147,12593,work,3,False,4,1070,1070,1359066,escort,13.381201,13,WALK,1.916676460871733 +10872536,33147,12593,work,4,False,4,560,1070,1359066,home,,13,DRIVEALONEFREE,-0.3973368603427345 +11957177,36454,13797,shopping,1,True,1,725,580,1494647,shopping,,13,SHARED2FREE,1.245325815705339 +11957181,36454,13797,shopping,1,False,2,761,725,1494647,shopping,15.764612,17,SHARED2FREE,1.1709751359944465 +11957182,36454,13797,shopping,2,False,2,580,761,1494647,home,,17,WALK,1.3392441443099874 +11957185,36454,13797,shopping,1,True,1,729,580,1494648,shopping,,18,WALK,-1.7031208771076614 +11957189,36454,13797,shopping,1,False,3,553,729,1494648,shopping,7.264665,18,WALK,-2.3051359687348576 +11957190,36454,13797,shopping,2,False,3,627,553,1494648,shopping,12.17949,18,WALK,0.05826922968555868 +11957191,36454,13797,shopping,3,False,3,580,627,1494648,home,,18,WALK,0.2535935405248986 +11957441,36455,13797,othdiscr,1,True,1,621,580,1494680,othdiscr,,16,DRIVEALONEFREE,0.18231278935857287 +11957445,36455,13797,othdiscr,1,False,3,988,621,1494680,othmaint,12.894039,21,DRIVEALONEFREE,0.21971831979581868 +11957446,36455,13797,othdiscr,2,False,3,715,988,1494680,social,12.710035,21,DRIVEALONEFREE,0.3418978355121365 +11957447,36455,13797,othdiscr,3,False,3,580,715,1494680,home,,21,DRIVEALONEFREE,0.30871307816540006 +11957553,36455,13797,work,1,True,1,555,580,1494694,work,,8,WALK,0.36344162230807797 +11957557,36455,13797,work,1,False,1,580,555,1494694,home,,12,WALK,0.3634848083682532 +13679289,41705,15777,eatout,1,True,1,1070,645,1709911,eatout,,9,DRIVEALONEFREE,0.3322583142495994 +13679293,41705,15777,eatout,1,False,1,645,1070,1709911,home,,15,SHARED2FREE,0.27985792727880493 +13679569,41706,15777,atwork,1,True,1,988,763,1709946,atwork,,12,WALK,-1.7814320077183017 +13679573,41706,15777,atwork,1,False,1,763,988,1709946,work,,16,WALK,-1.7814859079623158 +13679881,41706,15777,work,1,True,1,763,645,1709985,work,,8,DRIVEALONEFREE,0.07885921116729155 +13679885,41706,15777,work,1,False,2,1058,763,1709985,escort,12.955116,18,DRIVEALONEFREE,-0.23041784565166287 +13679886,41706,15777,work,2,False,2,645,1058,1709985,home,,19,DRIVEALONEFREE,-0.2819678531977345 +16411585,50035,18261,eatout,1,True,1,986,757,2051448,eatout,,16,BIKE,-0.7708543027540019 +16411589,50035,18261,eatout,1,False,2,676,986,2051448,escort,11.3688545,17,BIKE,-0.572314452735261 +16411590,50035,18261,eatout,2,False,2,757,676,2051448,home,,18,BIKE,0.6522763792928915 +16411729,50035,18261,school,1,True,1,919,757,2051466,school,,7,SHARED3FREE,-0.6140707263184741 +16411733,50035,18261,school,1,False,1,757,919,2051466,home,,13,DRIVEALONEFREE,-0.609643906803177 +16411745,50035,18261,shopping,1,True,1,991,757,2051468,shopping,,19,DRIVEALONEFREE,-0.4638089931309043 +16411749,50035,18261,shopping,1,False,1,757,991,2051468,home,,19,DRIVEALONEFREE,-0.456275824133683 +16412033,50036,18261,othmaint,1,True,1,874,757,2051504,othmaint,,7,TAXI,0.1889530517174145 +16412037,50036,18261,othmaint,1,False,1,757,874,2051504,home,,14,TAXI,0.16575320067330487 +16412449,50037,18261,work,1,True,1,1070,757,2051556,work,,7,DRIVEALONEFREE,-1.031094050528322 +16412453,50037,18261,work,1,False,1,757,1070,2051556,home,,16,DRIVEALONEFREE,-1.4470757107903176 +18151113,55338,19758,school,1,True,1,684,794,2268889,school,,7,WALK,-2.0063019658534276 +18151117,55338,19758,school,1,False,1,794,684,2268889,home,,10,WALK,-2.0063009106059084 +18151505,55339,19758,work,1,True,1,699,794,2268938,work,,5,WALK,-0.9284068819077078 +18151509,55339,19758,work,1,False,1,794,699,2268938,home,,16,WALK,-0.9284025287613823 +18990529,57897,20552,work,1,True,1,1070,829,2373816,work,,7,DRIVEALONEFREE,-0.8344257981984122 +18990533,57897,20552,work,1,False,1,829,1070,2373816,home,,20,DRIVEALONEFREE,-1.0366342952208958 +18990577,57898,20552,atwork,1,True,1,1070,904,2373822,atwork,,13,DRIVEALONEFREE,-0.27336926401076295 +18990581,57898,20552,atwork,1,False,1,904,1070,2373822,work,,14,DRIVEALONEFREE,-0.31175675978403394 +18990857,57898,20552,work,1,True,1,904,829,2373857,work,,11,DRIVEALONEFREE,-0.29059303362589683 +18990861,57898,20552,work,1,False,2,733,904,2373857,escort,11.759231,15,DRIVEALONEFREE,-0.17145224345062954 +18990862,57898,20552,work,2,False,2,829,733,2373857,home,,15,DRIVEALONEFREE,-0.13240928473806365 +18991185,57899,20552,work,1,True,1,687,829,2373898,work,,7,WALK,0.778418687259057 +18991189,57899,20552,work,1,False,1,829,687,2373898,home,,17,WALK,0.778424341749372 +18991537,57901,20552,atwork,1,True,1,990,706,2373942,atwork,,17,DRIVEALONEFREE,-0.009257198946139073 +18991541,57901,20552,atwork,1,False,1,706,990,2373942,work,,17,DRIVEALONEFREE,-0.016307475472911604 +18991841,57901,20552,work,1,True,1,706,829,2373980,work,,6,DRIVEALONEFREE,0.3008027167432845 +18991845,57901,20552,work,1,False,1,829,706,2373980,home,,12,DRIVEALONEFREE,0.2793884692543497 +18991849,57901,20552,work,1,True,2,712,829,2373981,othmaint,11.487295,15,DRIVEALONEFREE,-0.1906824640633464 +18991850,57901,20552,work,2,True,2,706,712,2373981,work,,16,DRIVEALONEFREE,0.2908894717624455 +18991853,57901,20552,work,1,False,1,829,706,2373981,home,,21,DRIVEALONEFREE,-0.21341173473575703 +20510417,62531,21869,school,1,True,1,943,900,2563802,school,,20,SHARED3FREE,-1.977211598694743 +20510421,62531,21869,school,1,False,1,900,943,2563802,home,,21,SHARED3FREE,-1.9596697588901777 +20510569,62532,21869,escort,1,True,1,647,900,2563821,escort,,6,SHARED2FREE,0.33462553060012734 +20510573,62532,21869,escort,1,False,1,900,647,2563821,home,,7,DRIVEALONEFREE,0.32064355812764017 +20510897,62533,21869,escort,1,True,1,695,900,2563862,escort,,5,SHARED3FREE,0.7144256053113908 +20510901,62533,21869,escort,1,False,4,996,695,2563862,shopping,13.912361,6,SHARED3FREE,0.8532067248808497 +20510902,62533,21869,escort,2,False,4,565,996,2563862,eatout,14.433976,6,SHARED3FREE,0.5414028726483867 +20510903,62533,21869,escort,3,False,4,1099,565,2563862,escort,14.882904,6,SHARED2FREE,0.5359149726020607 +20510904,62533,21869,escort,4,False,4,900,1099,2563862,home,,6,SHARED3FREE,0.8790959610035525 +20510905,62533,21869,escort,1,True,1,518,900,2563863,escort,,11,SHARED2FREE,0.022893992151203555 +20510909,62533,21869,escort,1,False,1,900,518,2563863,home,,11,SHARED2FREE,0.028638067201913687 +20510913,62533,21869,escort,1,True,1,844,900,2563864,escort,,14,SHARED2FREE,0.5953832464492131 +20510917,62533,21869,escort,1,False,1,900,844,2563864,home,,14,DRIVEALONEFREE,0.5982622478864028 +20511025,62533,21869,othdiscr,1,True,1,1070,900,2563878,othdiscr,,11,DRIVEALONEFREE,-0.1485066802108276 +20511029,62533,21869,othdiscr,1,False,1,900,1070,2563878,home,,12,DRIVEALONEFREE,-0.15348358194172215 +20511401,62534,21869,school,1,True,1,916,900,2563925,school,,8,WALK,-1.2930940202465637 +20511405,62534,21869,school,1,False,2,1082,916,2563925,shopping,7.8453183,9,SHARED2FREE,-1.5272294074732495 +20511406,62534,21869,school,2,False,2,900,1082,2563925,home,,9,WALK,1.1927962748590575 +22303745,67999,23619,escort,1,True,1,767,973,2787968,escort,,13,TNC_SHARED,0.1443338109842195 +22303749,67999,23619,escort,1,False,3,1070,767,2787968,eatout,12.697642,13,TNC_SINGLE,0.07988923600554106 +22303750,67999,23619,escort,2,False,3,1078,1070,2787968,escort,13.449805,13,TNC_SINGLE,0.7932467753193667 +22303751,67999,23619,escort,3,False,3,973,1078,2787968,home,,13,TNC_SINGLE,0.2276571099569908 +22303961,67999,23619,social,1,True,1,913,973,2787995,social,,17,WALK,-0.5148523504791808 +22303965,67999,23619,social,1,False,1,973,913,2787995,home,,20,WALK,-0.5150737184987781 +22304313,68000,23619,work,1,True,2,713,973,2788039,social,13.328311,7,SHARED2FREE,1.199888883674413 +22304314,68000,23619,work,2,True,2,1044,713,2788039,work,,8,DRIVEALONEFREE,0.20884911487460672 +22304317,68000,23619,work,1,False,2,506,1044,2788039,escort,14.330656,16,DRIVEALONEFREE,0.21452370815564967 +22304318,68000,23619,work,2,False,2,973,506,2788039,home,,19,SHARED2FREE,0.35306820841158415 +25904705,78977,26897,school,1,True,1,984,1081,3238088,school,,7,WALK,-0.4851985662306339 +25904709,78977,26897,school,1,False,1,1081,984,3238088,home,,14,WALK,-0.4852015217806304 +25905145,78979,26897,atwork,1,True,1,877,871,3238143,atwork,,9,WALK,-0.43405427476028074 +25905149,78979,26897,atwork,1,False,1,871,877,3238143,work,,11,WALK,-0.4342532950095454 +25905425,78979,26897,work,1,True,1,871,1081,3238178,work,,7,DRIVEALONEFREE,-0.21969701053725457 +25905429,78979,26897,work,1,False,1,1081,871,3238178,home,,18,DRIVEALONEFREE,-0.22317405833461496 +421021769,1283602,435012,work,1,True,1,1078,521,52627721,work,,8,DRIVEALONEFREE,-0.6750290142575426 +421021773,1283602,435012,work,1,False,2,560,1078,52627721,social,10.565217,14,DRIVEALONEFREE,-1.1907988454552887 +421021774,1283602,435012,work,2,False,2,521,560,52627721,home,,18,DRIVEALONEFREE,0.3055102301973978 +421108753,1283868,435278,eatout,1,True,2,1077,537,52638594,escort,14.320986,18,SHARED2FREE,-0.4350484631322457 +421108754,1283868,435278,eatout,2,True,2,1070,1077,52638594,eatout,,19,WALK,1.8006997660624806 +421108757,1283868,435278,eatout,1,False,1,537,1070,52638594,home,,21,SHARED2FREE,-0.49284538018699126 +421108889,1283868,435278,atwork,1,True,1,713,923,52638611,atwork,,16,DRIVEALONEFREE,-0.3206205648632515 +421108893,1283868,435278,atwork,1,False,1,923,713,52638611,work,,16,DRIVEALONEFREE,-0.30825543567593605 +421109017,1283868,435278,work,1,True,1,923,537,52638627,work,,9,DRIVEALONEFREE,-0.4470365744030762 +421109021,1283868,435278,work,1,False,2,579,923,52638627,work,11.985498,18,DRIVEALONEFREE,-0.45756848470905015 +421109022,1283868,435278,work,2,False,2,537,579,52638627,home,,18,DRIVEALONEFREE,0.022454544805313177 +421134601,1283946,435356,work,1,True,1,1005,523,52641825,work,,7,TNC_SINGLE,0.23663495333802967 +421134605,1283946,435356,work,1,False,1,523,1005,52641825,home,,18,TNC_SINGLE,0.133962425836884 +421348457,1284598,436008,work,1,True,1,551,562,52668557,work,,9,DRIVEALONEFREE,0.41482595505656983 +421348461,1284598,436008,work,1,False,1,562,551,52668557,home,,19,WALK,0.41482595505656983 +421878553,1286215,437625,atwork,1,True,1,1077,606,52734819,atwork,,10,DRIVEALONEFREE,-0.7443646785209397 +421878557,1286215,437625,atwork,1,False,1,606,1077,52734819,work,,13,DRIVEALONEFREE,-0.7600790932736868 +421878833,1286215,437625,work,1,True,1,606,656,52734854,work,,8,DRIVEALONEFREE,0.15855288808550738 +421878837,1286215,437625,work,1,False,3,934,606,52734854,othmaint,11.777501,17,DRIVEALONEFREE,-0.3271213767064194 +421878838,1286215,437625,work,2,False,3,730,934,52734854,othmaint,11.3140745,19,DRIVEALONEFREE,-0.1603647616823828 +421878839,1286215,437625,work,3,False,3,656,730,52734854,home,,19,WALK,0.16754237004219738 +423180353,1290184,441594,atwork,1,True,1,763,952,52897544,atwork,,11,DRIVEALONEFREE,-0.32097718137503994 +423180357,1290184,441594,atwork,1,False,1,952,763,52897544,work,,11,DRIVEALONEFREE,-0.308231300173503 +423180401,1290184,441594,eatout,1,True,1,946,1096,52897550,eatout,,20,DRIVEALONEFREE,-0.3549315699458884 +423180405,1290184,441594,eatout,1,False,2,1070,946,52897550,othmaint,11.981711,21,DRIVEALONEFREE,-0.6416413252299465 +423180406,1290184,441594,eatout,2,False,2,1096,1070,52897550,home,,21,DRIVEALONEFREE,0.5643804552711672 +423180553,1290184,441594,othdiscr,1,True,1,684,1096,52897569,othdiscr,,6,DRIVEALONEFREE,-0.054686106821676114 +423180557,1290184,441594,othdiscr,1,False,1,1096,684,52897569,home,,6,DRIVEALONEFREE,0.02546362680931599 +423180561,1290184,441594,othdiscr,1,True,1,762,1096,52897570,othdiscr,,21,DRIVEALONEFREE,-0.12826945310930996 +423180565,1290184,441594,othdiscr,1,False,1,1096,762,52897570,home,,22,DRIVEALONEFREE,-0.06385219068929958 +423180569,1290184,441594,othdiscr,1,True,2,1100,1096,52897571,eatout,11.933645,23,DRIVEALONEFREE,0.25460049444017807 +423180570,1290184,441594,othdiscr,2,True,2,725,1100,52897571,othdiscr,,23,DRIVEALONEFREE,0.15947563937372808 +423180573,1290184,441594,othdiscr,1,False,1,1096,725,52897571,home,,23,DRIVEALONEFREE,0.03984571997026273 +423180665,1290184,441594,work,1,True,1,952,1096,52897583,work,,9,DRIVEALONEFREE,-0.3390675752029656 +423180669,1290184,441594,work,1,False,2,1039,952,52897583,shopping,10.197688,19,DRIVEALONEFREE,0.04659305978780507 +423180670,1290184,441594,work,2,False,2,1096,1039,52897583,home,,20,DRIVEALONEFREE,-0.22006802386802082 +423325361,1290626,442036,atwork,1,True,1,1070,1070,52915670,atwork,,10,WALK,2.475344398627153 +423325365,1290626,442036,atwork,1,False,2,1070,1070,52915670,othmaint,20.94461,11,WALK,2.475344398627153 +423325366,1290626,442036,atwork,2,False,2,1070,1070,52915670,work,,11,WALK,2.475344398627153 +423325641,1290626,442036,work,1,True,2,1070,1100,52915705,work,16.471159,8,TAXI,0.5701727823879232 +423325642,1290626,442036,work,2,True,2,1070,1070,52915705,work,,8,WALK,2.0467518077554527 +423325645,1290626,442036,work,1,False,1,1100,1070,52915705,home,,18,WALK_LRF,0.798449501313428 +611033041,1862905,721960,othdiscr,1,True,1,975,960,76379130,othdiscr,,15,SHARED2FREE,0.8489306658693022 +611033045,1862905,721960,othdiscr,1,False,1,960,975,76379130,home,,21,DRIVEALONEFREE,0.8562669397756795 +611033369,1862906,721960,othdiscr,1,True,1,980,960,76379171,othdiscr,,11,SHARED2FREE,0.8131565508977017 +611033373,1862906,721960,othdiscr,1,False,2,596,980,76379171,eatout,14.574261,16,SHARED2FREE,0.6396850391241985 +611033374,1862906,721960,othdiscr,2,False,2,960,596,76379171,home,,16,SHARED2FREE,0.45683921107139835 +647572569,1974306,760593,othdiscr,1,True,1,564,509,80946571,othdiscr,,8,SHARED3FREE,0.9844823176047666 +647572573,1974306,760593,othdiscr,1,False,1,509,564,80946571,home,,16,SHARED3FREE,0.9905529876590623 +647572729,1974307,760593,atwork,1,True,1,853,696,80946591,atwork,,12,TNC_SINGLE,0.558880095631497 +647572733,1974307,760593,atwork,1,False,1,696,853,80946591,work,,13,TNC_SINGLE,0.4374826660603407 +647573009,1974307,760593,work,1,True,2,670,509,80946626,shopping,7.846782,9,WALK,-0.2893554824550794 +647573010,1974307,760593,work,2,True,2,696,670,80946626,work,,10,WALK,-0.5062069887028856 +647573013,1974307,760593,work,1,False,1,509,696,80946626,home,,18,WALK,-1.3939945137991185 +647573097,1974308,760593,escort,1,True,1,716,509,80946637,escort,,5,SHARED3FREE,0.8932804409891216 +647573101,1974308,760593,escort,1,False,1,509,716,80946637,home,,5,DRIVEALONEFREE,0.8953863138451676 +648387521,1976791,761445,escort,1,True,1,908,613,81048440,escort,,13,DRIVEALONEFREE,0.4328362377795291 +648387525,1976791,761445,escort,1,False,1,613,908,81048440,home,,13,SHARED2FREE,0.42852804729578203 +648387809,1976792,761445,atwork,1,True,1,517,517,81048476,atwork,,9,DRIVEALONEFREE,0.424692568411527 +648387813,1976792,761445,atwork,1,False,1,517,517,81048476,work,,9,DRIVEALONEFREE,0.424692568411527 +648388065,1976792,761445,social,1,True,1,831,613,81048508,social,,14,WALK,-1.207803332065376 +648388069,1976792,761445,social,1,False,1,613,831,81048508,home,,19,WALK,-1.2077283870682878 +648388089,1976792,761445,work,1,True,1,517,613,81048511,work,,5,DRIVEALONEFREE,-0.14314373582115136 +648388093,1976792,761445,work,1,False,1,613,517,81048511,home,,12,DRIVEALONEFREE,-0.1608731894314201 +649042753,1978788,762159,social,1,True,1,739,961,81130344,social,,8,SHARED3FREE,0.8737334926181418 +649042757,1978788,762159,social,1,False,1,961,739,81130344,home,,20,SHARED3FREE,0.870208545905065 +649043193,1978790,762159,escort,1,True,1,992,961,81130399,escort,,8,SHARED2FREE,0.5349665117215714 +649043197,1978790,762159,escort,1,False,1,961,992,81130399,home,,8,SHARED2FREE,0.5205938584654705 +649043433,1978790,762159,work,1,True,1,681,961,81130429,work,,8,DRIVEALONEFREE,-0.23724932749335306 +649043437,1978790,762159,work,1,False,1,961,681,81130429,home,,17,DRIVEALONEFREE,-0.22922899735997304 +649043761,1978791,762159,work,1,True,1,1070,961,81130470,work,,7,DRIVEALONEFREE,-0.2212113034329037 +649043765,1978791,762159,work,1,False,1,961,1070,81130470,home,,17,SHARED2FREE,-0.44936442701724144 +819359665,2498047,922602,school,1,True,1,737,730,102419958,school,,9,WALK,-0.034487644429644575 +819359669,2498047,922602,school,1,False,1,730,737,102419958,home,,16,WALK,-0.03405380538927606 +819360057,2498048,922602,work,1,True,1,722,730,102420007,work,,7,WALK,-0.5757010138694136 +819360061,2498048,922602,work,1,False,1,730,722,102420007,home,,16,WALK,-0.5758022615450138 +819360385,2498049,922602,work,1,True,1,755,730,102420048,work,,6,WALK,-2.1334401182407383 +819360389,2498049,922602,work,1,False,1,730,755,102420048,home,,16,WALK,-2.14642409582644 +860079225,2622192,952720,school,1,True,1,1004,1025,107509903,school,,7,WALK,-0.2623158950918999 +860079229,2622192,952720,school,1,False,1,1025,1004,107509903,home,,14,WALK,-0.2623158950918999 +860079377,2622193,952720,escort,1,True,1,773,1025,107509922,escort,,6,DRIVEALONEFREE,0.5629145900985214 +860079381,2622193,952720,escort,1,False,1,1025,773,107509922,home,,6,SHARED2FREE,0.5769090248299181 +860079529,2622193,952720,othmaint,1,True,1,550,1025,107509941,othmaint,,8,DRIVEALONEFREE,0.4954921210447033 +860079533,2622193,952720,othmaint,1,False,1,1025,550,107509941,home,,15,TAXI,0.5175746060032502 +860079897,2622194,952720,shopping,1,True,1,989,1025,107509987,shopping,,9,WALK,0.7145051809777091 +860079901,2622194,952720,shopping,1,False,1,1025,989,107509987,home,,11,WALK,0.7145046724011186 +860080273,2622195,952720,work,1,True,1,1021,1025,107510034,work,,8,DRIVEALONEFREE,0.15716016268583638 +860080277,2622195,952720,work,1,False,1,1025,1021,107510034,home,,17,DRIVEALONEFREE,0.15709169882202492 +933123249,2844887,1028031,work,1,True,1,796,846,116640406,work,,11,DRIVEALONEFREE,0.12762068821382538 +933123253,2844887,1028031,work,1,False,2,756,796,116640406,eatout,12.25646,20,DRIVEALONEFREE,-0.03647709997422193 +933123254,2844887,1028031,work,2,False,2,846,756,116640406,home,,21,DRIVEALONEFREE,-0.04899107659316716 +962301409,2933845,1048898,school,1,True,1,563,574,120287676,school,,12,WALK,0.6010774748532366 +962301413,2933845,1048898,school,1,False,1,574,563,120287676,home,,21,WALK,0.6011107780954241 +962301737,2933846,1048898,school,1,True,1,515,574,120287717,school,,8,WALK,-2.077872721293554 +962301741,2933846,1048898,school,1,False,1,574,515,120287717,home,,16,SHARED2FREE,-2.0830952831662595 +962302017,2933847,1048898,othdiscr,1,True,1,623,574,120287752,othdiscr,,7,TNC_SINGLE,0.16536386928459096 +962302021,2933847,1048898,othdiscr,1,False,2,877,623,120287752,eatout,12.331069,12,TAXI,0.10034848748228002 +962302022,2933847,1048898,othdiscr,2,False,2,574,877,120287752,home,,12,DRIVEALONEFREE,0.015704822984971067 +962302457,2933848,1048898,work,1,True,1,502,574,120287807,work,,6,DRIVEALONEFREE,-0.05432573197742724 +962302461,2933848,1048898,work,1,False,2,728,502,120287807,escort,12.163573,17,DRIVEALONEFREE,-0.09018878571767061 +962302462,2933848,1048898,work,2,False,2,574,728,120287807,home,,18,DRIVEALONEFREE,-0.19676785298448135 +1055052265,3216622,1148260,univ,1,True,1,1074,1076,131881533,univ,,14,WALK,-1.344631554321622 +1055052269,3216622,1148260,univ,1,False,3,1074,1074,131881533,univ,9.8536825,15,WALK,-0.5067482665291204 +1055052270,3216622,1148260,univ,2,False,3,1070,1074,131881533,escort,12.185525,15,WALK,-0.7310853671900902 +1055052271,3216622,1148260,univ,3,False,3,1076,1070,131881533,home,,15,WALK,1.4364023787928928 diff --git a/activitysim/examples/placeholder_sandag/test/regress/final_3_zone_tours.csv b/activitysim/examples/placeholder_sandag/test/regress/final_3_zone_tours.csv index 23434cccc2..a27174eec9 100644 --- a/activitysim/examples/placeholder_sandag/test/regress/final_3_zone_tours.csv +++ b/activitysim/examples/placeholder_sandag/test/regress/final_3_zone_tours.csv @@ -1,79 +1,78 @@ tour_id,person_id,tour_type,tour_type_count,tour_type_num,tour_num,tour_count,tour_category,number_of_participants,destination,origin,household_id,tdd,start,end,duration,composition,destination_logsum,tour_mode,mode_choice_logsum,od_atap,od_btap,od_path_set,do_atap,do_btap,do_path_set,atwork_subtour_frequency,parent_tour_id,stop_frequency,primary_purpose -1359025,33146,work,1,1,1,1,mandatory,1,1100.0,560.0,12593,12.0,5.0,17.0,12.0,,,DRIVEALONEFREE,1.0261372219238083,,,,,,,no_subtours,,2out_0in,work -1359066,33147,work,1,1,1,1,mandatory,1,1070.0,560.0,12593,61.0,8.0,15.0,7.0,,,WALK_TRANSIT,2.96683804953191,1500.0,1558.0,fastest,1558.0,1500.0,fastest,no_subtours,,0out_1in,work -1494647,36454,shopping,2,1,1,2,non_mandatory,1,725.0,580.0,13797,128.0,13.0,17.0,4.0,,11.688295162318989,DRIVEALONEFREE,0.185754815801926,,,,,,,,,0out_1in,shopping -1494648,36454,shopping,2,2,2,2,non_mandatory,1,729.0,580.0,13797,169.0,18.0,18.0,0.0,,11.689029568940938,DRIVEALONEFREE,0.443644139698639,,,,,,,,,1out_0in,shopping -1494680,36455,othdiscr,1,1,1,1,non_mandatory,1,623.0,580.0,13797,159.0,16.0,21.0,5.0,,13.239720693390336,SHARED2FREE,0.738377038878971,,,,,,,,,0out_2in,othdiscr -1494694,36455,work,1,1,1,1,mandatory,1,562.0,580.0,13797,58.0,8.0,12.0,4.0,,,SHARED3FREE,0.8851264867711228,,,,,,,no_subtours,,0out_0in,work -1709911,41705,eatout,1,1,1,1,non_mandatory,1,1070.0,645.0,15777,76.0,9.0,15.0,6.0,,12.070246786563017,WALK_TRANSIT,0.4629895150090414,1500.0,1611.0,fastest,1611.0,1500.0,shortest,,,0out_0in,eatout -1709950,41706,eat,1,1,1,1,atwork,1,989.0,854.0,15777,112.0,12.0,12.0,0.0,,18.50954665100095,WALK_TRANSIT,-0.6180823333381719,1666.0,1662.0,shortest,1748.0,1618.0,fastest,,1709985.0,0out_0in,atwork -1709985,41706,work,1,1,1,1,mandatory,1,854.0,645.0,15777,65.0,8.0,19.0,11.0,,,SHARED2FREE,0.613674174636675,,,,,,,eat,,0out_1in,work -2051448,50035,eatout,1,1,1,1,joint,3,1070.0,757.0,18261,156.0,16.0,18.0,2.0,mixed,11.85828127629614,SHARED3FREE,-5.832306485220664,,,,,,,,,1out_0in,eatout -2051466,50035,school,1,1,1,1,mandatory,1,919.0,757.0,18261,43.0,7.0,13.0,6.0,,,WALK_TRANSIT,1.2158966517539809,1618.0,1748.0,shortest,1748.0,1618.0,fastest,,,0out_0in,school -2051468,50035,shopping,1,1,1,1,non_mandatory,1,996.0,757.0,18261,175.0,19.0,19.0,0.0,,11.577466521097683,DRIVEALONEFREE,-0.1846581633337935,,,,,,,,,0out_0in,shopping -2051504,50036,othmaint,1,1,1,1,non_mandatory,1,877.0,757.0,18261,44.0,7.0,14.0,7.0,,12.412335687839308,SHARED2FREE,0.0818069109613803,,,,,,,,,0out_0in,othmaint -2051556,50037,work,1,1,1,1,mandatory,1,1070.0,757.0,18261,46.0,7.0,16.0,9.0,,,WALK_TRANSIT,1.2858135302346076,1500.0,1651.0,shortest,1651.0,1500.0,shortest,no_subtours,,0out_0in,work -2268889,55338,school,1,1,1,1,mandatory,1,699.0,794.0,19758,40.0,7.0,10.0,3.0,,,WALK_TRANSIT,1.426179634142238,1608.0,1652.0,cheapest,1652.0,1608.0,fastest,,,0out_0in,school -2268938,55339,work,1,1,1,1,mandatory,1,762.0,794.0,19758,11.0,5.0,16.0,11.0,,,DRIVEALONEFREE,0.6319219718888117,,,,,,,no_subtours,,0out_0in,work -2373816,57897,work,1,1,1,1,mandatory,1,1070.0,829.0,20552,50.0,7.0,20.0,13.0,,,WALK_TRANSIT,1.815253620726848,1500.0,1754.0,fastest,1754.0,1500.0,shortest,no_subtours,,0out_0in,work -2373818,57898,business,1,1,1,1,atwork,1,948.0,948.0,20552,101.0,11.0,13.0,2.0,,10.856328670591866,WALK,1.0849001811607144,,,,,,,,2373857.0,1out_0in,atwork -2373857,57898,work,1,1,1,1,mandatory,1,948.0,829.0,20552,105.0,11.0,17.0,6.0,,,DRIVEALONEFREE,0.2266199059946182,,,,,,,business1,,0out_1in,work -2373898,57899,work,1,1,1,1,mandatory,1,687.0,829.0,20552,47.0,7.0,17.0,10.0,,,WALK,1.4925958516365831,,,,,,,no_subtours,,0out_0in,work -2373980,57901,work,2,1,1,2,mandatory,1,708.0,829.0,20552,29.0,6.0,16.0,10.0,,,DRIVEALONEFREE,0.5654424599486602,,,,,,,no_subtours,,0out_0in,work -2373981,57901,work,2,2,2,2,mandatory,1,708.0,829.0,20552,172.0,18.0,21.0,3.0,,,SHARED2FREE,0.7217981727639898,,,,,,,no_subtours,,1out_0in,work -2563802,62531,school,1,1,1,1,mandatory,1,938.0,900.0,21869,180.0,20.0,20.0,0.0,,,WALK_TRANSIT,1.0406873179045706,1673.0,1633.0,shortest,1633.0,1673.0,fastest,,,0out_0in,school -2563821,62532,escort,1,1,1,1,non_mandatory,1,647.0,900.0,21869,20.0,6.0,7.0,1.0,,12.49976194812137,SHARED2FREE,-1.2299200748899617,,,,,,,,,0out_0in,escort -2563862,62533,escort,3,1,1,4,non_mandatory,1,695.0,900.0,21869,1.0,5.0,6.0,1.0,,12.55798878217292,SHARED3FREE,-1.221458591960288,,,,,,,,,0out_3in,escort -2563863,62533,escort,3,2,2,4,non_mandatory,1,518.0,900.0,21869,99.0,11.0,11.0,0.0,,12.502003827155296,SHARED3FREE,-1.66067049305692,,,,,,,,,0out_0in,escort -2563864,62533,escort,3,3,3,4,non_mandatory,1,844.0,900.0,21869,135.0,14.0,14.0,0.0,,12.530459798779978,SHARED2FREE,-1.4032965315757244,,,,,,,,,0out_0in,escort -2563878,62533,othdiscr,1,1,4,4,non_mandatory,1,1070.0,900.0,21869,100.0,11.0,12.0,1.0,,13.216136797297771,WALK_TRANSIT,1.0118533227067692,1500.0,73.0,fastest,73.0,1500.0,shortest,,,0out_0in,othdiscr -2563925,62534,school,1,1,1,1,mandatory,1,793.0,900.0,21869,55.0,8.0,9.0,1.0,,,WALK_TRANSIT,-0.605105363231441,1748.0,73.0,shortest,73.0,1748.0,shortest,,,0out_0in,school -2787968,67999,escort,1,1,1,2,non_mandatory,1,767.0,973.0,23619,124.0,13.0,13.0,0.0,,12.970512717862755,SHARED3FREE,-0.7145790466379958,,,,,,,,,0out_2in,escort -2787995,67999,social,1,1,2,2,non_mandatory,1,988.0,973.0,23619,165.0,17.0,20.0,3.0,,12.777797143559908,WALK,1.394261833722182,,,,,,,,,0out_0in,social -2788039,68000,work,1,1,1,1,mandatory,1,1070.0,973.0,23619,51.0,7.0,21.0,14.0,,,WALK_TRANSIT,1.4899770411640134,1500.0,1588.0,shortest,1588.0,1500.0,fastest,no_subtours,,1out_0in,work -3238088,78977,school,1,1,1,1,mandatory,1,984.0,1081.0,26897,44.0,7.0,14.0,7.0,,,WALK,1.0391837359866254,,,,,,,,,0out_0in,school -3238143,78979,eat,1,1,1,1,atwork,1,1070.0,897.0,26897,72.0,9.0,11.0,2.0,,18.50007970900582,DRIVEALONEFREE,-0.2864770605138423,,,,,,,,3238178.0,0out_0in,atwork -3238178,78979,work,1,1,1,1,mandatory,1,897.0,1081.0,26897,48.0,7.0,18.0,11.0,,,DRIVEALONEFREE,0.5747485518145101,,,,,,,eat,,0out_0in,work -52627721,1283602,work,1,1,1,1,mandatory,1,1078.0,521.0,435012,64.0,8.0,18.0,10.0,,,WALK_TRANSIT,5.710266024459359,1584.0,1238.0,shortest,1238.0,1584.0,cheapest,no_subtours,,0out_0in,work -52638588,1283868,business,1,1,1,1,atwork,1,1070.0,1070.0,435278,112.0,12.0,12.0,0.0,,19.38117027561944,WALK,0.6487888616012731,,,,,,,,52638627.0,0out_0in,atwork -52638594,1283868,eatout,1,1,1,1,non_mandatory,1,1070.0,537.0,435278,172.0,18.0,21.0,3.0,,11.755720899317405,WALK_TRANSIT,0.6406085296048486,1500.0,1558.0,fastest,1558.0,1500.0,shortest,,,0out_1in,eatout -52638627,1283868,work,1,1,1,1,mandatory,1,1070.0,537.0,435278,79.0,9.0,18.0,9.0,,,WALK_TRANSIT,1.206681423660966,1500.0,1558.0,shortest,1558.0,1500.0,shortest,business1,,0out_0in,work -52641825,1283946,work,1,1,1,1,mandatory,1,1070.0,523.0,435356,49.0,7.0,19.0,12.0,,,WALK_TRANSIT,1.1978831616543206,1500.0,1604.0,cheapest,1604.0,1500.0,shortest,no_subtours,,0out_0in,work -52668557,1284598,work,1,1,1,1,mandatory,1,553.0,562.0,436008,80.0,9.0,19.0,10.0,,,DRIVEALONEFREE,0.7312528815647517,,,,,,,no_subtours,,0out_0in,work -52734819,1286215,eat,1,1,1,1,atwork,1,1077.0,606.0,437625,88.0,10.0,13.0,3.0,,18.149830935609163,WALK_TRANSIT,-0.8798916104770301,1500.0,1562.0,shortest,1562.0,1500.0,shortest,,52734854.0,0out_0in,atwork -52734854,1286215,work,1,1,1,1,mandatory,1,606.0,656.0,437625,65.0,8.0,19.0,11.0,,,DRIVEALONEFREE,0.8985332322364314,,,,,,,eat,,0out_2in,work -52897544,1290184,business,1,1,1,1,atwork,1,732.0,977.0,441594,99.0,11.0,11.0,0.0,,18.610370946991612,SHARED2FREE,0.4316303329603235,,,,,,,,52897583.0,0out_0in,atwork -52897550,1290184,eatout,1,1,4,4,non_mandatory,1,949.0,1096.0,441594,184.0,21.0,21.0,0.0,,11.904706692046382,SHARED2FREE,-0.1665710953231831,,,,,,,,,0out_1in,eatout -52897569,1290184,othdiscr,3,1,1,4,non_mandatory,1,684.0,1096.0,441594,2.0,5.0,7.0,2.0,,12.918529902113356,DRIVEALONEFREE,0.2317787046274241,,,,,,,,,0out_0in,othdiscr -52897570,1290184,othdiscr,3,2,2,4,non_mandatory,1,762.0,1096.0,441594,185.0,21.0,22.0,1.0,,12.959861282775266,WALK_TRANSIT,0.2522845990878585,1748.0,1516.0,fastest,1516.0,1748.0,fastest,,,0out_0in,othdiscr -52897571,1290184,othdiscr,3,3,3,4,non_mandatory,1,725.0,1096.0,441594,189.0,23.0,23.0,0.0,,13.001363665570665,WALK,0.3854413463230495,,,,,,,,,0out_0in,othdiscr -52897583,1290184,work,1,1,1,1,mandatory,1,977.0,1096.0,441594,82.0,9.0,21.0,12.0,,,DRIVEALONEFREE,-0.0423820492992088,,,,,,,business1,,0out_1in,work -52915705,1290626,work,1,1,1,1,mandatory,1,1070.0,1100.0,442036,64.0,8.0,18.0,10.0,,,WALK_TRANSIT,1.4095982213888998,1500.0,1584.0,cheapest,1584.0,1500.0,fastest,no_subtours,,1out_0in,work -76379130,1862905,othdiscr,1,1,1,1,non_mandatory,1,986.0,960.0,721960,151.0,15.0,21.0,6.0,,12.957346522442643,WALK_TRANSIT,0.3212522660901984,1618.0,1762.0,fastest,1762.0,1618.0,fastest,,,0out_0in,othdiscr -76379171,1862906,othdiscr,1,1,1,1,non_mandatory,1,992.0,960.0,721960,104.0,11.0,16.0,5.0,,12.9976638956563,SHARED3FREE,0.3134675853717786,,,,,,,,,0out_1in,othdiscr -80946571,1974306,othdiscr,1,1,1,1,non_mandatory,1,564.0,509.0,760593,62.0,8.0,16.0,8.0,,13.57415728090829,WALK,1.2942555315594169,,,,,,,,,0out_0in,othdiscr -80946591,1974307,eat,1,1,1,1,atwork,1,913.0,739.0,760593,113.0,12.0,13.0,1.0,,18.622653002301583,SHARED3FREE,-0.7672328773497228,,,,,,,,80946626.0,0out_0in,atwork -80946626,1974307,work,1,1,1,1,mandatory,1,739.0,509.0,760593,79.0,9.0,18.0,9.0,,,WALK_TRANSIT,0.6201259698287482,1748.0,1603.0,fastest,1665.0,1662.0,cheapest,eat,,1out_1in,work -80946637,1974308,escort,1,1,1,1,non_mandatory,1,716.0,509.0,760593,0.0,5.0,5.0,0.0,,12.671799977139084,SHARED3FREE,-0.8180421450990675,,,,,,,,,0out_0in,escort -81048440,1976791,escort,1,1,1,1,non_mandatory,1,908.0,613.0,761445,124.0,13.0,13.0,0.0,,12.82099191373982,SHARED2FREE,-0.9528167859302504,,,,,,,,,0out_0in,escort -81048476,1976792,eat,1,1,1,1,atwork,1,517.0,517.0,761445,70.0,9.0,9.0,0.0,,17.910037480926228,SHARED2FREE,0.7181216397530412,,,,,,,,81048511.0,0out_0in,atwork -81048478,1976792,eatout,1,1,1,1,non_mandatory,1,648.0,613.0,761445,130.0,13.0,19.0,6.0,,11.955235691700665,DRIVEALONEFREE,1.3804088714164349,,,,,,,,,0out_0in,eatout -81048511,1976792,work,1,1,1,1,mandatory,1,517.0,613.0,761445,7.0,5.0,12.0,7.0,,,WALK_TRANSIT,0.5051019359212883,1681.0,1238.0,fastest,1238.0,1681.0,fastest,eat,,0out_0in,work -81130344,1978788,social,1,1,1,1,non_mandatory,1,763.0,961.0,762159,66.0,8.0,20.0,12.0,,12.393316604403664,SHARED2FREE,1.1233925360416968,,,,,,,,,0out_0in,social -81130399,1978790,escort,1,1,1,1,non_mandatory,1,992.0,961.0,762159,54.0,8.0,8.0,0.0,,12.58022321361327,SHARED2FREE,-0.9361958649826466,,,,,,,,,0out_0in,escort -81130429,1978790,work,1,1,1,1,mandatory,1,672.0,961.0,762159,63.0,8.0,17.0,9.0,,,DRIVEALONEFREE,0.7319548477569705,,,,,,,no_subtours,,0out_0in,work -81130470,1978791,work,1,1,1,1,mandatory,1,1070.0,961.0,762159,47.0,7.0,17.0,10.0,,,WALK_TRANSIT,1.223231536517431,1500.0,1762.0,shortest,1762.0,1500.0,shortest,no_subtours,,0out_0in,work -102419958,2498047,school,1,1,1,1,mandatory,1,865.0,730.0,922602,77.0,9.0,16.0,7.0,,,WALK_TRANSIT,4.515115660099912,1698.0,1608.0,shortest,1608.0,1698.0,shortest,,,0out_0in,school -102420007,2498048,work,1,1,1,1,mandatory,1,735.0,730.0,922602,46.0,7.0,16.0,9.0,,,WALK_TRANSIT,2.37613290143943,1592.0,1612.0,shortest,1612.0,1592.0,shortest,no_subtours,,0out_0in,work -102420048,2498049,work,1,1,1,1,mandatory,1,763.0,730.0,922602,29.0,6.0,16.0,10.0,,,WALK,2.522839906360249,,,,,,,no_subtours,,0out_0in,work -107509903,2622192,school,1,1,1,1,mandatory,1,995.0,1025.0,952720,44.0,7.0,14.0,7.0,,,WALK_TRANSIT,0.7240925397746193,1664.0,1666.0,fastest,1666.0,1664.0,fastest,,,0out_0in,school -107509922,2622193,escort,1,1,1,2,non_mandatory,1,773.0,1025.0,952720,19.0,6.0,6.0,0.0,,12.840047018957565,SHARED3FREE,-0.7318852867656347,,,,,,,,,0out_0in,escort -107509941,2622193,othmaint,1,1,2,2,non_mandatory,1,550.0,1025.0,952720,61.0,8.0,15.0,7.0,,12.86750275018236,WALK_TRANSIT,0.3002279969733954,1238.0,1666.0,fastest,1666.0,1238.0,cheapest,,,0out_0in,othmaint -107509987,2622194,shopping,1,1,1,1,non_mandatory,1,989.0,1025.0,952720,72.0,9.0,11.0,2.0,,11.974087902319576,SHARED3FREE,0.5678792550825605,,,,,,,,,0out_0in,shopping -107510034,2622195,work,1,1,1,1,mandatory,1,1021.0,1025.0,952720,63.0,8.0,17.0,9.0,,,DRIVEALONEFREE,0.8640931880479102,,,,,,,no_subtours,,0out_0in,work -116640406,2844887,work,1,1,1,1,mandatory,1,845.0,846.0,1028031,111.0,11.0,23.0,12.0,,,WALK_TRANSIT,0.9802398468480144,1730.0,1695.0,shortest,1695.0,1730.0,cheapest,no_subtours,,0out_0in,work -120287676,2933845,school,1,1,1,1,mandatory,1,666.0,574.0,1048898,121.0,12.0,21.0,9.0,,,WALK,-0.0074452524495141,,,,,,,,,0out_0in,school -120287717,2933846,school,1,1,1,1,mandatory,1,515.0,574.0,1048898,62.0,8.0,16.0,8.0,,,SHARED2FREE,-1.016021399976598,,,,,,,,,0out_0in,school -120287752,2933847,othdiscr,1,1,1,1,non_mandatory,1,657.0,574.0,1048898,42.0,7.0,12.0,5.0,,13.002449945790438,WALK_TRANSIT,0.8636594074657715,1238.0,1559.0,shortest,1559.0,1238.0,shortest,,,0out_1in,othdiscr -120287807,2933848,work,1,1,1,1,mandatory,1,502.0,574.0,1048898,31.0,6.0,18.0,12.0,,,WALK_TRANSIT,0.3885773372794717,1333.0,1559.0,fastest,1559.0,1333.0,fastest,no_subtours,,0out_0in,work -131881533,3216622,school,1,1,1,1,mandatory,1,1074.0,1076.0,1148260,136.0,14.0,15.0,1.0,,,WALK_TRANSIT,10.725339744357893,1500.0,1516.0,fastest,1516.0,1500.0,fastest,,,0out_0in,univ +1359025,33146,work,1,1,1,1,mandatory,1,608.0,68.0,12593,12.0,5.0,17.0,12.0,,,DRIVEALONEFREE,1.0261372219238083,,,,,,,no_subtours,,2out_0in,work +1359066,33147,work,1,1,1,1,mandatory,1,578.0,68.0,12593,61.0,8.0,15.0,7.0,,,WALK_TRANSIT,2.96683804953191,1500.0,1558.0,fastest,1558.0,1500.0,fastest,no_subtours,,0out_1in,work +1494647,36454,shopping,2,1,1,2,non_mandatory,1,233.0,88.0,13797,128.0,13.0,17.0,4.0,,11.594151670549255,DRIVEALONEFREE,0.054207142123309696,,,,,,,,,0out_1in,shopping +1494648,36454,shopping,2,2,2,2,non_mandatory,1,237.0,88.0,13797,169.0,18.0,18.0,0.0,,11.643159811140553,DRIVEALONEFREE,0.28212638921439376,,,,,,,,,1out_0in,shopping +1494680,36455,othdiscr,1,1,1,1,non_mandatory,1,131.0,88.0,13797,159.0,16.0,21.0,5.0,,13.153011009083952,DRIVEALONEFREE,0.527847608692733,,,,,,,,,0out_2in,othdiscr +1494694,36455,work,1,1,1,1,mandatory,1,70.0,88.0,13797,58.0,8.0,12.0,4.0,,,WALK,0.9189166731374394,,,,,,,no_subtours,,0out_0in,work +1709911,41705,eatout,1,1,1,1,non_mandatory,1,578.0,153.0,15777,76.0,9.0,15.0,6.0,,12.194280696346672,WALK_TRANSIT,0.4629895150090414,1500.0,1611.0,fastest,1611.0,1500.0,fastest,,,0out_0in,eatout +1709950,41706,eat,1,1,1,1,atwork,1,497.0,362.0,15777,112.0,12.0,12.0,0.0,,18.60610008578578,WALK_TRANSIT,-0.8459966967797716,1666.0,1662.0,shortest,1662.0,1666.0,shortest,,1709985.0,0out_0in,atwork +1709985,41706,work,1,1,1,1,mandatory,1,362.0,153.0,15777,65.0,8.0,19.0,11.0,,,SHARED2FREE,0.6600577445845757,,,,,,,eat,,0out_1in,work +2051448,50035,eatout,1,1,1,1,joint,3,578.0,265.0,18261,156.0,16.0,18.0,2.0,mixed,11.830383451121103,SHARED3FREE,-5.832306485220664,,,,,,,,,1out_0in,eatout +2051466,50035,school,1,1,1,1,mandatory,1,427.0,265.0,18261,43.0,7.0,13.0,6.0,,,WALK_TRANSIT,1.2158966517539807,1618.0,1748.0,shortest,1748.0,1618.0,shortest,,,0out_0in,school +2051468,50035,shopping,1,1,1,1,non_mandatory,1,504.0,265.0,18261,175.0,19.0,19.0,0.0,,11.561744392977129,DRIVEALONEFREE,-0.18465816333379353,,,,,,,,,0out_0in,shopping +2051504,50036,othmaint,1,1,1,1,non_mandatory,1,385.0,265.0,18261,44.0,7.0,14.0,7.0,,12.388911839471218,SHARED2FREE,0.07492591448018238,,,,,,,,,0out_0in,othmaint +2051556,50037,work,1,1,1,1,mandatory,1,578.0,265.0,18261,46.0,7.0,16.0,9.0,,,WALK_TRANSIT,1.2858135302346076,1500.0,1651.0,shortest,1651.0,1500.0,fastest,no_subtours,,0out_0in,work +2268889,55338,school,1,1,1,1,mandatory,1,207.0,302.0,19758,40.0,7.0,10.0,3.0,,,WALK_TRANSIT,1.4625238798878268,1608.0,1652.0,cheapest,1652.0,1608.0,shortest,,,0out_0in,school +2268938,55339,work,1,1,1,1,mandatory,1,254.0,302.0,19758,11.0,5.0,16.0,11.0,,,DRIVEALONEFREE,0.899253309377072,,,,,,,no_subtours,,0out_0in,work +2373816,57897,work,1,1,1,1,mandatory,1,578.0,337.0,20552,50.0,7.0,20.0,13.0,,,WALK_TRANSIT,1.8152536207268481,1500.0,1754.0,fastest,1754.0,1500.0,fastest,no_subtours,,0out_0in,work +2373818,57898,business,1,1,1,1,atwork,1,456.0,456.0,20552,101.0,11.0,13.0,2.0,,10.851045587482702,WALK,0.8933720529791014,,,,,,,,2373857.0,1out_0in,atwork +2373857,57898,work,1,1,1,1,mandatory,1,456.0,337.0,20552,105.0,11.0,17.0,6.0,,,DRIVEALONEFREE,0.22661990599461826,,,,,,,business1,,0out_1in,work +2373898,57899,work,1,1,1,1,mandatory,1,195.0,337.0,20552,47.0,7.0,17.0,10.0,,,WALK,1.4775079963774336,,,,,,,no_subtours,,0out_0in,work +2373980,57901,work,2,1,1,2,mandatory,1,216.0,337.0,20552,30.0,6.0,17.0,11.0,,,DRIVEALONEFREE,0.5418945770189671,,,,,,,no_subtours,,0out_0in,work +2373981,57901,work,2,2,2,2,mandatory,1,216.0,337.0,20552,175.0,19.0,19.0,0.0,,,SHARED2FREE,0.7907156266134642,,,,,,,no_subtours,,1out_0in,work +2563802,62531,school,1,1,1,1,mandatory,1,442.0,408.0,21869,180.0,20.0,20.0,0.0,,,WALK_TRANSIT,0.9438904813360477,1682.0,1633.0,shortest,1633.0,1682.0,fastest,,,0out_0in,school +2563821,62532,escort,1,1,1,1,non_mandatory,1,155.0,408.0,21869,20.0,6.0,7.0,1.0,,12.523318372217064,SHARED2FREE,-1.2299200748899615,,,,,,,,,0out_0in,escort +2563862,62533,escort,3,1,1,4,non_mandatory,1,203.0,408.0,21869,1.0,5.0,6.0,1.0,,12.561374920193618,SHARED3FREE,-1.221458591960288,,,,,,,,,0out_3in,escort +2563863,62533,escort,3,2,2,4,non_mandatory,1,26.0,408.0,21869,99.0,11.0,11.0,0.0,,12.495641230564226,SHARED3FREE,-1.66067049305692,,,,,,,,,0out_0in,escort +2563864,62533,escort,3,3,3,4,non_mandatory,1,352.0,408.0,21869,135.0,14.0,14.0,0.0,,12.525318452376542,SHARED2FREE,-1.4032965315757244,,,,,,,,,0out_0in,escort +2563878,62533,othdiscr,1,1,4,4,non_mandatory,1,578.0,408.0,21869,100.0,11.0,12.0,1.0,,13.255455507967595,WALK_TRANSIT,1.0602578337312834,1500.0,73.0,fastest,73.0,1500.0,shortest,,,0out_0in,othdiscr +2563925,62534,school,1,1,1,1,mandatory,1,301.0,408.0,21869,55.0,8.0,9.0,1.0,,,WALK_TRANSIT,-0.605105363231441,1748.0,73.0,shortest,73.0,1748.0,fastest,,,0out_0in,school +2787968,67999,escort,1,1,1,2,non_mandatory,1,269.0,481.0,23619,124.0,13.0,13.0,0.0,,12.995994495540678,SHARED3FREE,-0.6687154899908945,,,,,,,,,0out_2in,escort +2787995,67999,social,1,1,2,2,non_mandatory,1,496.0,481.0,23619,165.0,17.0,20.0,3.0,,12.759866800729268,WALK,1.4266623253551904,,,,,,,,,0out_0in,social +2788039,68000,work,1,1,1,1,mandatory,1,578.0,481.0,23619,51.0,7.0,21.0,14.0,,,WALK_TRANSIT,1.4899770411640134,1500.0,1588.0,shortest,1588.0,1500.0,shortest,no_subtours,,1out_0in,work +3238088,78977,school,1,1,1,1,mandatory,1,492.0,589.0,26897,44.0,7.0,14.0,7.0,,,WALK_TRANSIT,0.8204920756912042,1618.0,1584.0,shortest,1584.0,1618.0,cheapest,,,0out_0in,school +3238143,78979,eat,1,1,1,1,atwork,1,578.0,405.0,26897,72.0,9.0,11.0,2.0,,18.599435534464497,DRIVEALONEFREE,-0.2744746714261923,,,,,,,,3238178.0,0out_0in,atwork +3238178,78979,work,1,1,1,1,mandatory,1,405.0,589.0,26897,48.0,7.0,18.0,11.0,,,DRIVEALONEFREE,0.5976010496480395,,,,,,,eat,,0out_0in,work +52627721,1283602,work,1,1,1,1,mandatory,1,586.0,29.0,435012,64.0,8.0,18.0,10.0,,,WALK_TRANSIT,5.710266024459359,1584.0,1238.0,shortest,1608.0,1584.0,fastest,no_subtours,,0out_0in,work +52638588,1283868,business,1,1,1,1,atwork,1,578.0,578.0,435278,112.0,12.0,12.0,0.0,,19.462265786986954,WALK,0.6243659427953904,,,,,,,,52638627.0,0out_0in,atwork +52638594,1283868,eatout,1,1,1,1,non_mandatory,1,578.0,45.0,435278,172.0,18.0,21.0,3.0,,11.758619181788253,WALK_TRANSIT,0.6406085296048486,1500.0,1558.0,fastest,1558.0,1500.0,shortest,,,0out_1in,eatout +52638627,1283868,work,1,1,1,1,mandatory,1,578.0,45.0,435278,79.0,9.0,18.0,9.0,,,WALK_TRANSIT,1.206681423660966,1500.0,1558.0,shortest,1558.0,1500.0,shortest,business1,,0out_0in,work +52641825,1283946,work,1,1,1,1,mandatory,1,578.0,31.0,435356,49.0,7.0,19.0,12.0,,,WALK_TRANSIT,1.1978831616543206,1500.0,1604.0,cheapest,1604.0,1500.0,shortest,no_subtours,,0out_0in,work +52668557,1284598,work,1,1,1,1,mandatory,1,61.0,70.0,436008,80.0,9.0,19.0,10.0,,,DRIVEALONEFREE,0.7713829364314543,,,,,,,no_subtours,,0out_0in,work +52734819,1286215,eat,1,1,1,1,atwork,1,585.0,114.0,437625,88.0,10.0,13.0,3.0,,17.98372758878566,WALK_TRANSIT,-0.8798916104770301,1500.0,1562.0,shortest,1562.0,1500.0,cheapest,,52734854.0,0out_0in,atwork +52734854,1286215,work,1,1,1,1,mandatory,1,114.0,164.0,437625,65.0,8.0,19.0,11.0,,,DRIVEALONEFREE,0.8500450570238016,,,,,,,eat,,0out_2in,work +52897544,1290184,business,1,1,1,1,atwork,1,240.0,485.0,441594,99.0,11.0,11.0,0.0,,18.649706295366613,WALK,0.7361914154560637,,,,,,,,52897583.0,0out_0in,atwork +52897550,1290184,eatout,1,1,4,4,non_mandatory,1,347.0,604.0,441594,184.0,21.0,21.0,0.0,,11.9769068256242,SHARED2FREE,-0.10785441309565102,,,,,,,,,0out_1in,eatout +52897569,1290184,othdiscr,3,1,1,4,non_mandatory,1,22.0,604.0,441594,2.0,5.0,7.0,2.0,,13.340592587029276,DRIVEALONEFREE,0.9729989937926249,,,,,,,,,0out_0in,othdiscr +52897570,1290184,othdiscr,3,2,2,4,non_mandatory,1,241.0,604.0,441594,185.0,21.0,22.0,1.0,,13.194495693775464,BIKE,0.24171550040319387,,,,,,,,,0out_0in,othdiscr +52897571,1290184,othdiscr,3,3,3,4,non_mandatory,1,161.0,604.0,441594,189.0,23.0,23.0,0.0,,13.296474125502314,WALK_TRANSIT,0.4347637705876943,1238.0,1516.0,fastest,1516.0,1238.0,fastest,,,0out_1in,othdiscr +52897583,1290184,work,1,1,1,1,mandatory,1,485.0,604.0,441594,82.0,9.0,21.0,12.0,,,DRIVEALONEFREE,-0.05454824120553145,,,,,,,business1,,0out_1in,work +52915705,1290626,work,1,1,1,1,mandatory,1,578.0,608.0,442036,64.0,8.0,18.0,10.0,,,WALK_TRANSIT,1.4241213520203073,1500.0,1584.0,cheapest,1584.0,1500.0,fastest,no_subtours,,1out_0in,work +76379130,1862905,othdiscr,1,1,1,1,non_mandatory,1,494.0,468.0,721960,151.0,15.0,21.0,6.0,,13.046748957111205,BIKE,0.6336063394695053,,,,,,,,,0out_0in,othdiscr +76379171,1862906,othdiscr,1,1,1,1,non_mandatory,1,500.0,468.0,721960,104.0,11.0,16.0,5.0,,13.068781711323547,WALK,0.6246154614956402,,,,,,,,,0out_0in,othdiscr +80946571,1974306,othdiscr,1,1,1,1,non_mandatory,1,72.0,17.0,760593,62.0,8.0,16.0,8.0,,13.609717599377104,WALK,1.2877790624306358,,,,,,,,,0out_0in,othdiscr +80946591,1974307,eat,1,1,1,1,atwork,1,324.0,204.0,760593,113.0,12.0,13.0,1.0,,18.711722700372903,SHARED3FREE,-1.1971419362787026,,,,,,,,80946626.0,0out_0in,atwork +80946626,1974307,work,1,1,1,1,mandatory,1,204.0,17.0,760593,79.0,9.0,18.0,9.0,,,WALK_TRANSIT,0.9146373054676561,1748.0,1603.0,shortest,1603.0,1748.0,shortest,eat,,1out_1in,work +80946637,1974308,escort,1,1,1,1,non_mandatory,1,224.0,17.0,760593,0.0,5.0,5.0,0.0,,12.668506375964688,SHARED3FREE,-0.8073126044826399,,,,,,,,,0out_0in,escort +81048440,1976791,escort,1,1,1,1,non_mandatory,1,483.0,121.0,761445,124.0,13.0,13.0,0.0,,12.714018122401603,SHARED2FREE,-0.7186631667509052,,,,,,,,,0out_0in,escort +81048508,1976792,social,1,1,1,1,non_mandatory,1,494.0,121.0,761445,140.0,14.0,19.0,5.0,,12.159236859223718,DRIVEALONEFREE,0.5721200907291696,,,,,,,,,0out_0in,social +81048511,1976792,work,1,1,1,1,mandatory,1,25.0,121.0,761445,7.0,5.0,12.0,7.0,,,WALK_TRANSIT,0.4973097110196049,1681.0,1238.0,fastest,1238.0,1681.0,fastest,no_subtours,,0out_0in,work +81130344,1978788,social,1,1,1,1,non_mandatory,1,271.0,469.0,762159,66.0,8.0,20.0,12.0,,12.430877422027173,SHARED2FREE,1.1233925360416968,,,,,,,,,0out_0in,social +81130399,1978790,escort,1,1,1,1,non_mandatory,1,500.0,469.0,762159,54.0,8.0,8.0,0.0,,12.604750255865364,SHARED2FREE,-0.7891363476365268,,,,,,,,,0out_0in,escort +81130429,1978790,work,1,1,1,1,mandatory,1,180.0,469.0,762159,63.0,8.0,17.0,9.0,,,DRIVEALONEFREE,0.7319548477569705,,,,,,,no_subtours,,0out_0in,work +81130470,1978791,work,1,1,1,1,mandatory,1,578.0,469.0,762159,47.0,7.0,17.0,10.0,,,WALK_TRANSIT,1.2232315365174313,1500.0,1762.0,shortest,1762.0,1500.0,shortest,no_subtours,,0out_0in,work +102419958,2498047,school,1,1,1,1,mandatory,1,432.0,238.0,922602,77.0,9.0,16.0,7.0,,,WALK_TRANSIT,4.547714059900562,1673.0,1608.0,shortest,1608.0,1673.0,fastest,,,0out_0in,school +102420007,2498048,work,1,1,1,1,mandatory,1,243.0,238.0,922602,46.0,7.0,16.0,9.0,,,WALK_TRANSIT,2.571846550195381,1592.0,1612.0,shortest,1612.0,1592.0,shortest,no_subtours,,0out_0in,work +102420048,2498049,work,1,1,1,1,mandatory,1,271.0,238.0,922602,29.0,6.0,16.0,10.0,,,WALK,2.3529244532782645,,,,,,,no_subtours,,0out_0in,work +107509903,2622192,school,1,1,1,1,mandatory,1,501.0,533.0,952720,44.0,7.0,14.0,7.0,,,WALK_TRANSIT,0.7689513133515259,1664.0,1666.0,fastest,1666.0,1664.0,shortest,,,0out_0in,school +107509922,2622193,escort,1,1,1,2,non_mandatory,1,281.0,533.0,952720,19.0,6.0,6.0,0.0,,13.120197714243572,WALK,-0.6633544461803778,,,,,,,,,0out_0in,escort +107509941,2622193,othmaint,1,1,2,2,non_mandatory,1,58.0,533.0,952720,61.0,8.0,15.0,7.0,,13.296284550786224,WALK_TRANSIT,0.6229120227429399,1238.0,1666.0,fastest,1666.0,1238.0,shortest,,,0out_0in,othmaint +107509987,2622194,shopping,1,1,1,1,non_mandatory,1,504.0,533.0,952720,74.0,9.0,13.0,4.0,,12.281026585137033,SHARED3FREE,0.8228058752894183,,,,,,,,,0out_0in,shopping +107510034,2622195,work,1,1,1,1,mandatory,1,578.0,533.0,952720,63.0,8.0,17.0,9.0,,,WALK_TRANSIT,1.4377271736671302,1500.0,1666.0,fastest,1666.0,1500.0,shortest,no_subtours,,0out_0in,work +116640406,2844887,work,1,1,1,1,mandatory,1,385.0,354.0,1028031,108.0,11.0,20.0,9.0,,,WALK_TRANSIT,0.6883028572332556,1754.0,1695.0,shortest,1748.0,1754.0,fastest,no_subtours,,0out_0in,work +120287676,2933845,school,1,1,1,1,mandatory,1,197.0,82.0,1048898,121.0,12.0,21.0,9.0,,,WALK_TRANSIT,-0.05133861963844917,1608.0,1559.0,shortest,1559.0,1608.0,fastest,,,0out_0in,school +120287717,2933846,school,1,1,1,1,mandatory,1,23.0,82.0,1048898,62.0,8.0,16.0,8.0,,,SHARED2FREE,-1.0238153653421556,,,,,,,,,0out_0in,school +120287752,2933847,othdiscr,1,1,1,1,non_mandatory,1,165.0,82.0,1048898,42.0,7.0,12.0,5.0,,12.971588539704879,WALK_TRANSIT,0.8435740874298338,1238.0,1559.0,shortest,1559.0,1238.0,fastest,,,0out_1in,othdiscr +120287807,2933848,work,1,1,1,1,mandatory,1,10.0,82.0,1048898,31.0,6.0,18.0,12.0,,,WALK_TRANSIT,0.3435861763727887,1333.0,1559.0,fastest,1559.0,1333.0,shortest,no_subtours,,0out_0in,work +131881533,3216622,school,1,1,1,1,mandatory,1,582.0,584.0,1148260,136.0,14.0,15.0,1.0,,,WALK_TRANSIT,10.730879630807463,1500.0,1516.0,fastest,1516.0,1500.0,shortest,,,0out_0in,univ diff --git a/activitysim/examples/placeholder_sandag/test/regress/final_3_zone_tours_sh.csv b/activitysim/examples/placeholder_sandag/test/regress/final_3_zone_tours_sh.csv new file mode 100644 index 0000000000..2016fde26b --- /dev/null +++ b/activitysim/examples/placeholder_sandag/test/regress/final_3_zone_tours_sh.csv @@ -0,0 +1,79 @@ +tour_id,person_id,tour_type,tour_type_count,tour_type_num,tour_num,tour_count,tour_category,number_of_participants,destination,origin,household_id,tdd,start,end,duration,composition,destination_logsum,tour_mode,mode_choice_logsum,od_atap,od_btap,od_path_set,do_atap,do_btap,do_path_set,atwork_subtour_frequency,parent_tour_id,stop_frequency,primary_purpose +1359025,33146,work,1,1,1,1,mandatory,1,1100,560,12593,12.0,5.0,17.0,12.0,,,DRIVEALONEFREE,1.0261372219238083,,,,,,,no_subtours,,2out_0in,work +1359066,33147,work,1,1,1,1,mandatory,1,1070,560,12593,61.0,8.0,15.0,7.0,,,WALK_TRANSIT,2.96683804953191,1500.0,1558.0,fastest,1558.0,1500.0,fastest,no_subtours,,0out_1in,work +1494647,36454,shopping,2,1,1,2,non_mandatory,1,725,580,13797,128.0,13.0,17.0,4.0,,11.688295162318989,DRIVEALONEFREE,0.18575481580192602,,,,,,,,,0out_1in,shopping +1494648,36454,shopping,2,2,2,2,non_mandatory,1,729,580,13797,169.0,18.0,18.0,0.0,,11.689029568940938,DRIVEALONEFREE,0.44364413969863903,,,,,,,,,1out_0in,shopping +1494680,36455,othdiscr,1,1,1,1,non_mandatory,1,623,580,13797,159.0,16.0,21.0,5.0,,13.239720693390337,SHARED2FREE,0.738377038878971,,,,,,,,,0out_2in,othdiscr +1494694,36455,work,1,1,1,1,mandatory,1,562,580,13797,58.0,8.0,12.0,4.0,,,SHARED3FREE,0.8851264867711228,,,,,,,no_subtours,,0out_0in,work +1709911,41705,eatout,1,1,1,1,non_mandatory,1,1070,645,15777,76.0,9.0,15.0,6.0,,12.070246786563015,WALK_TRANSIT,0.4629895150090414,1500.0,1611.0,fastest,1611.0,1500.0,fastest,,,0out_0in,eatout +1709950,41706,eat,1,1,1,1,atwork,1,989,854,15777,112.0,12.0,12.0,0.0,,18.50954665100095,WALK_TRANSIT,-0.6180823333381719,1666.0,1662.0,shortest,1662.0,1666.0,shortest,,1709985.0,0out_0in,atwork +1709985,41706,work,1,1,1,1,mandatory,1,854,645,15777,65.0,8.0,19.0,11.0,,,SHARED2FREE,0.613674174636675,,,,,,,eat,,0out_1in,work +2051448,50035,eatout,1,1,1,1,joint,3,1070,757,18261,156.0,16.0,18.0,2.0,mixed,11.85828127629614,SHARED3FREE,-5.832306485220664,,,,,,,,,1out_0in,eatout +2051466,50035,school,1,1,1,1,mandatory,1,919,757,18261,43.0,7.0,13.0,6.0,,,WALK_TRANSIT,1.2158966517539807,1618.0,1748.0,shortest,1748.0,1618.0,shortest,,,0out_0in,school +2051468,50035,shopping,1,1,1,1,non_mandatory,1,996,757,18261,175.0,19.0,19.0,0.0,,11.577466521097683,DRIVEALONEFREE,-0.18465816333379353,,,,,,,,,0out_0in,shopping +2051504,50036,othmaint,1,1,1,1,non_mandatory,1,877,757,18261,44.0,7.0,14.0,7.0,,12.412335687839308,SHARED2FREE,0.0818069109613803,,,,,,,,,0out_0in,othmaint +2051556,50037,work,1,1,1,1,mandatory,1,1070,757,18261,46.0,7.0,16.0,9.0,,,WALK_TRANSIT,1.2858135302346076,1500.0,1651.0,shortest,1651.0,1500.0,fastest,no_subtours,,0out_0in,work +2268889,55338,school,1,1,1,1,mandatory,1,699,794,19758,40.0,7.0,10.0,3.0,,,WALK_TRANSIT,1.4261796341422381,1608.0,1652.0,cheapest,1652.0,1608.0,shortest,,,0out_0in,school +2268938,55339,work,1,1,1,1,mandatory,1,762,794,19758,11.0,5.0,16.0,11.0,,,DRIVEALONEFREE,0.6319219718888117,,,,,,,no_subtours,,0out_0in,work +2373816,57897,work,1,1,1,1,mandatory,1,1070,829,20552,50.0,7.0,20.0,13.0,,,WALK_TRANSIT,1.8152536207268481,1500.0,1754.0,fastest,1754.0,1500.0,fastest,no_subtours,,0out_0in,work +2373818,57898,business,1,1,1,1,atwork,1,948,948,20552,101.0,11.0,13.0,2.0,,10.856328670591866,WALK,1.0849001811607144,,,,,,,,2373857.0,1out_0in,atwork +2373857,57898,work,1,1,1,1,mandatory,1,948,829,20552,105.0,11.0,17.0,6.0,,,DRIVEALONEFREE,0.22661990599461826,,,,,,,business1,,0out_1in,work +2373898,57899,work,1,1,1,1,mandatory,1,687,829,20552,47.0,7.0,17.0,10.0,,,WALK,1.4925958516365831,,,,,,,no_subtours,,0out_0in,work +2373980,57901,work,2,1,1,2,mandatory,1,708,829,20552,30.0,6.0,17.0,11.0,,,DRIVEALONEFREE,0.5418945770189671,,,,,,,no_subtours,,0out_0in,work +2373981,57901,work,2,2,2,2,mandatory,1,708,829,20552,175.0,19.0,19.0,0.0,,,SHARED2FREE,0.7907156266134642,,,,,,,no_subtours,,1out_0in,work +2563802,62531,school,1,1,1,1,mandatory,1,938,900,21869,180.0,20.0,20.0,0.0,,,WALK_TRANSIT,1.0406873179045706,1673.0,1633.0,shortest,1633.0,1673.0,fastest,,,0out_0in,school +2563821,62532,escort,1,1,1,1,non_mandatory,1,647,900,21869,20.0,6.0,7.0,1.0,,12.49976194812137,SHARED2FREE,-1.2299200748899615,,,,,,,,,0out_0in,escort +2563862,62533,escort,3,1,1,4,non_mandatory,1,695,900,21869,1.0,5.0,6.0,1.0,,12.55798878217292,SHARED3FREE,-1.221458591960288,,,,,,,,,0out_3in,escort +2563863,62533,escort,3,2,2,4,non_mandatory,1,518,900,21869,99.0,11.0,11.0,0.0,,12.502003827155296,SHARED3FREE,-1.66067049305692,,,,,,,,,0out_0in,escort +2563864,62533,escort,3,3,3,4,non_mandatory,1,844,900,21869,135.0,14.0,14.0,0.0,,12.530459798779978,SHARED2FREE,-1.4032965315757244,,,,,,,,,0out_0in,escort +2563878,62533,othdiscr,1,1,4,4,non_mandatory,1,1070,900,21869,100.0,11.0,12.0,1.0,,13.216136797297771,WALK_TRANSIT,1.0118533227067692,1500.0,73.0,fastest,73.0,1500.0,shortest,,,0out_0in,othdiscr +2563925,62534,school,1,1,1,1,mandatory,1,793,900,21869,55.0,8.0,9.0,1.0,,,WALK_TRANSIT,-0.605105363231441,1748.0,73.0,shortest,73.0,1748.0,fastest,,,0out_0in,school +2787968,67999,escort,1,1,1,2,non_mandatory,1,767,973,23619,124.0,13.0,13.0,0.0,,12.970512717862757,SHARED3FREE,-0.7145790466379958,,,,,,,,,0out_2in,escort +2787995,67999,social,1,1,2,2,non_mandatory,1,988,973,23619,165.0,17.0,20.0,3.0,,12.777797143559908,WALK,1.394261833722182,,,,,,,,,0out_0in,social +2788039,68000,work,1,1,1,1,mandatory,1,1070,973,23619,51.0,7.0,21.0,14.0,,,WALK_TRANSIT,1.4899770411640134,1500.0,1588.0,shortest,1588.0,1500.0,shortest,no_subtours,,1out_0in,work +3238088,78977,school,1,1,1,1,mandatory,1,984,1081,26897,44.0,7.0,14.0,7.0,,,WALK,1.0391837359866254,,,,,,,,,0out_0in,school +3238143,78979,eat,1,1,1,1,atwork,1,1070,897,26897,72.0,9.0,11.0,2.0,,18.50007970900582,DRIVEALONEFREE,-0.28647706051384236,,,,,,,,3238178.0,0out_0in,atwork +3238178,78979,work,1,1,1,1,mandatory,1,897,1081,26897,48.0,7.0,18.0,11.0,,,DRIVEALONEFREE,0.5747485518145101,,,,,,,eat,,0out_0in,work +52627721,1283602,work,1,1,1,1,mandatory,1,1078,521,435012,64.0,8.0,18.0,10.0,,,WALK_TRANSIT,5.710266024459359,1584.0,1238.0,shortest,1608.0,1584.0,fastest,no_subtours,,0out_0in,work +52638588,1283868,business,1,1,1,1,atwork,1,1070,1070,435278,112.0,12.0,12.0,0.0,,19.38117027561944,WALK,0.6487888616012731,,,,,,,,52638627.0,0out_0in,atwork +52638594,1283868,eatout,1,1,1,1,non_mandatory,1,1070,537,435278,172.0,18.0,21.0,3.0,,11.755720899317405,WALK_TRANSIT,0.6406085296048486,1500.0,1558.0,fastest,1558.0,1500.0,shortest,,,0out_1in,eatout +52638627,1283868,work,1,1,1,1,mandatory,1,1070,537,435278,79.0,9.0,18.0,9.0,,,WALK_TRANSIT,1.206681423660966,1500.0,1558.0,shortest,1558.0,1500.0,shortest,business1,,0out_0in,work +52641825,1283946,work,1,1,1,1,mandatory,1,1070,523,435356,49.0,7.0,19.0,12.0,,,WALK_TRANSIT,1.1978831616543206,1500.0,1604.0,cheapest,1604.0,1500.0,shortest,no_subtours,,0out_0in,work +52668557,1284598,work,1,1,1,1,mandatory,1,553,562,436008,80.0,9.0,19.0,10.0,,,DRIVEALONEFREE,0.7312528815647517,,,,,,,no_subtours,,0out_0in,work +52734819,1286215,eat,1,1,1,1,atwork,1,1077,606,437625,88.0,10.0,13.0,3.0,,18.149830935609163,WALK_TRANSIT,-0.8798916104770301,1500.0,1562.0,shortest,1562.0,1500.0,cheapest,,52734854.0,0out_0in,atwork +52734854,1286215,work,1,1,1,1,mandatory,1,606,656,437625,65.0,8.0,19.0,11.0,,,DRIVEALONEFREE,0.8985332322364314,,,,,,,eat,,0out_2in,work +52897544,1290184,business,1,1,1,1,atwork,1,732,977,441594,99.0,11.0,11.0,0.0,,18.610370946991615,SHARED2FREE,0.43163033296032355,,,,,,,,52897583.0,0out_0in,atwork +52897550,1290184,eatout,1,1,4,4,non_mandatory,1,949,1096,441594,184.0,21.0,21.0,0.0,,11.904706692046382,SHARED2FREE,-0.16657109532318312,,,,,,,,,0out_1in,eatout +52897569,1290184,othdiscr,3,1,1,4,non_mandatory,1,684,1096,441594,2.0,5.0,7.0,2.0,,12.918529902113356,DRIVEALONEFREE,0.23177870462742411,,,,,,,,,0out_0in,othdiscr +52897570,1290184,othdiscr,3,2,2,4,non_mandatory,1,762,1096,441594,185.0,21.0,22.0,1.0,,12.959861282775266,WALK_TRANSIT,0.25228459908785855,1748.0,1516.0,fastest,1516.0,1748.0,shortest,,,0out_0in,othdiscr +52897571,1290184,othdiscr,3,3,3,4,non_mandatory,1,725,1096,441594,189.0,23.0,23.0,0.0,,13.001363665570663,WALK,0.38544134632304955,,,,,,,,,0out_0in,othdiscr +52897583,1290184,work,1,1,1,1,mandatory,1,977,1096,441594,82.0,9.0,21.0,12.0,,,DRIVEALONEFREE,-0.04238204929920889,,,,,,,business1,,0out_1in,work +52915705,1290626,work,1,1,1,1,mandatory,1,1070,1100,442036,64.0,8.0,18.0,10.0,,,WALK_TRANSIT,1.4095982213888998,1500.0,1584.0,cheapest,1584.0,1500.0,fastest,no_subtours,,1out_0in,work +76379130,1862905,othdiscr,1,1,1,1,non_mandatory,1,986,960,721960,151.0,15.0,21.0,6.0,,12.957346522442645,WALK_TRANSIT,0.32125226609019847,1618.0,1762.0,fastest,1762.0,1618.0,cheapest,,,0out_0in,othdiscr +76379171,1862906,othdiscr,1,1,1,1,non_mandatory,1,992,960,721960,104.0,11.0,16.0,5.0,,12.9976638956563,SHARED3FREE,0.3134675853717786,,,,,,,,,0out_1in,othdiscr +80946571,1974306,othdiscr,1,1,1,1,non_mandatory,1,564,509,760593,62.0,8.0,16.0,8.0,,13.57415728090829,WALK,1.2942555315594169,,,,,,,,,0out_0in,othdiscr +80946591,1974307,eat,1,1,1,1,atwork,1,913,739,760593,113.0,12.0,13.0,1.0,,18.622653002301586,SHARED3FREE,-0.7672328773497228,,,,,,,,80946626.0,0out_0in,atwork +80946626,1974307,work,1,1,1,1,mandatory,1,739,509,760593,79.0,9.0,18.0,9.0,,,WALK_TRANSIT,0.6201259698287482,1748.0,1603.0,fastest,1603.0,1748.0,fastest,eat,,1out_1in,work +80946637,1974308,escort,1,1,1,1,non_mandatory,1,716,509,760593,0.0,5.0,5.0,0.0,,12.671799977139083,SHARED3FREE,-0.8180421450990675,,,,,,,,,0out_0in,escort +81048440,1976791,escort,1,1,1,1,non_mandatory,1,908,613,761445,124.0,13.0,13.0,0.0,,12.820991913739821,SHARED2FREE,-0.9528167859302505,,,,,,,,,0out_0in,escort +81048476,1976792,eat,1,1,1,1,atwork,1,517,517,761445,70.0,9.0,9.0,0.0,,17.910037480926228,SHARED2FREE,0.7181216397530412,,,,,,,,81048511.0,0out_0in,atwork +81048478,1976792,eatout,1,1,1,1,non_mandatory,1,648,613,761445,130.0,13.0,19.0,6.0,,11.955235691700665,DRIVEALONEFREE,1.3804088714164349,,,,,,,,,0out_0in,eatout +81048511,1976792,work,1,1,1,1,mandatory,1,517,613,761445,7.0,5.0,12.0,7.0,,,WALK_TRANSIT,0.5051019359212883,1681.0,1238.0,fastest,1238.0,1681.0,fastest,eat,,0out_0in,work +81130344,1978788,social,1,1,1,1,non_mandatory,1,763,961,762159,66.0,8.0,20.0,12.0,,12.393316604403664,SHARED2FREE,1.1233925360416968,,,,,,,,,0out_0in,social +81130399,1978790,escort,1,1,1,1,non_mandatory,1,992,961,762159,54.0,8.0,8.0,0.0,,12.58022321361327,SHARED2FREE,-0.9361958649826466,,,,,,,,,0out_0in,escort +81130429,1978790,work,1,1,1,1,mandatory,1,672,961,762159,63.0,8.0,17.0,9.0,,,DRIVEALONEFREE,0.7319548477569705,,,,,,,no_subtours,,0out_0in,work +81130470,1978791,work,1,1,1,1,mandatory,1,1070,961,762159,47.0,7.0,17.0,10.0,,,WALK_TRANSIT,1.2232315365174313,1500.0,1762.0,shortest,1762.0,1500.0,shortest,no_subtours,,0out_0in,work +102419958,2498047,school,1,1,1,1,mandatory,1,865,730,922602,77.0,9.0,16.0,7.0,,,WALK_TRANSIT,4.515115660099912,1698.0,1608.0,shortest,1608.0,1698.0,fastest,,,0out_0in,school +102420007,2498048,work,1,1,1,1,mandatory,1,735,730,922602,46.0,7.0,16.0,9.0,,,WALK_TRANSIT,2.37613290143943,1592.0,1612.0,shortest,1612.0,1592.0,shortest,no_subtours,,0out_0in,work +102420048,2498049,work,1,1,1,1,mandatory,1,763,730,922602,29.0,6.0,16.0,10.0,,,WALK,2.522839906360249,,,,,,,no_subtours,,0out_0in,work +107509903,2622192,school,1,1,1,1,mandatory,1,995,1025,952720,44.0,7.0,14.0,7.0,,,WALK_TRANSIT,0.7240925397746193,1664.0,1666.0,fastest,1666.0,1664.0,shortest,,,0out_0in,school +107509922,2622193,escort,1,1,1,2,non_mandatory,1,773,1025,952720,19.0,6.0,6.0,0.0,,12.840047018957563,SHARED3FREE,-0.7318852867656347,,,,,,,,,0out_0in,escort +107509941,2622193,othmaint,1,1,2,2,non_mandatory,1,550,1025,952720,61.0,8.0,15.0,7.0,,12.867502750182359,WALK_TRANSIT,0.3002279969733954,1238.0,1666.0,fastest,1666.0,1238.0,shortest,,,0out_0in,othmaint +107509987,2622194,shopping,1,1,1,1,non_mandatory,1,989,1025,952720,72.0,9.0,11.0,2.0,,11.974087902319576,SHARED3FREE,0.5678792550825605,,,,,,,,,0out_0in,shopping +107510034,2622195,work,1,1,1,1,mandatory,1,1021,1025,952720,63.0,8.0,17.0,9.0,,,DRIVEALONEFREE,0.8640931880479102,,,,,,,no_subtours,,0out_0in,work +116640406,2844887,work,1,1,1,1,mandatory,1,845,846,1028031,111.0,11.0,23.0,12.0,,,WALK_TRANSIT,0.9802398468480145,1730.0,1695.0,shortest,1695.0,1748.0,fastest,no_subtours,,0out_0in,work +120287676,2933845,school,1,1,1,1,mandatory,1,666,574,1048898,121.0,12.0,21.0,9.0,,,WALK,-0.007445252449514138,,,,,,,,,0out_0in,school +120287717,2933846,school,1,1,1,1,mandatory,1,515,574,1048898,62.0,8.0,16.0,8.0,,,SHARED2FREE,-1.016021399976598,,,,,,,,,0out_0in,school +120287752,2933847,othdiscr,1,1,1,1,non_mandatory,1,657,574,1048898,42.0,7.0,12.0,5.0,,13.002449945790438,WALK_TRANSIT,0.8636594074657715,1238.0,1559.0,shortest,1559.0,1238.0,fastest,,,0out_1in,othdiscr +120287807,2933848,work,1,1,1,1,mandatory,1,502,574,1048898,31.0,6.0,18.0,12.0,,,WALK_TRANSIT,0.38857733727947175,1333.0,1559.0,fastest,1559.0,1333.0,shortest,no_subtours,,0out_0in,work +131881533,3216622,school,1,1,1,1,mandatory,1,1074,1076,1148260,136.0,14.0,15.0,1.0,,,WALK_TRANSIT,10.725339744357893,1500.0,1516.0,fastest,1516.0,1500.0,shortest,,,0out_0in,univ diff --git a/activitysim/examples/placeholder_sandag/test/regress/final_3_zone_trips.csv b/activitysim/examples/placeholder_sandag/test/regress/final_3_zone_trips.csv index 03a015e705..8c93f8ec82 100644 --- a/activitysim/examples/placeholder_sandag/test/regress/final_3_zone_trips.csv +++ b/activitysim/examples/placeholder_sandag/test/regress/final_3_zone_trips.csv @@ -1,185 +1,183 @@ trip_id,person_id,household_id,primary_purpose,trip_num,outbound,trip_count,destination,origin,tour_id,purpose,destination_logsum,depart,trip_mode,mode_choice_logsum,atap,btap,path_set -10872201,33146,12593,work,1,True,3,879,560,1359025,escort,11.70885284520791,5,DRIVEALONEFREE,-0.3548746577733072,,, -10872202,33146,12593,work,2,True,3,989,879,1359025,escort,12.022433281644073,7,DRIVEALONEFREE,-0.361684345054043,,, -10872203,33146,12593,work,3,True,3,1100,989,1359025,work,,7,DRIVEALONEFREE,0.1801428827020189,,, -10872205,33146,12593,work,1,False,1,560,1100,1359025,home,,17,DRIVEALONEFREE,-0.368339088809198,,, -10872529,33147,12593,work,1,True,1,1070,560,1359066,work,,8,WALK_TRANSIT,2.7567263588191,1500.0,1558.0,shortest -10872533,33147,12593,work,1,False,2,942,1070,1359066,escort,19.449635104296835,15,DRIVEALONEFREE,1.4771881293112104,,, -10872534,33147,12593,work,2,False,2,560,942,1359066,home,,15,SHARED2FREE,1.0374930881113709,,, -11957177,36454,13797,shopping,1,True,1,725,580,1494647,shopping,,13,DRIVEALONEFREE,0.1717448995480592,,, -11957181,36454,13797,shopping,1,False,2,803,725,1494647,shopping,12.016029658018422,17,TAXI,0.0352267728440656,,, -11957182,36454,13797,shopping,2,False,2,580,803,1494647,home,,17,DRIVEALONEFREE,0.07359791064631,,, -11957185,36454,13797,shopping,1,True,2,687,580,1494648,escort,13.593599471819292,18,WALK,0.7788942047635835,,, -11957186,36454,13797,shopping,2,True,2,729,687,1494648,shopping,,18,DRIVEALONEFREE,0.489210171281661,,, -11957189,36454,13797,shopping,1,False,1,580,729,1494648,home,,18,TAXI,0.2313076964301194,,, -11957441,36455,13797,othdiscr,1,True,1,623,580,1494680,othdiscr,,16,DRIVEALONEFREE,0.7744647479734814,,, -11957445,36455,13797,othdiscr,1,False,3,784,623,1494680,othmaint,15.224351954248831,21,SHARED2FREE,0.6723499661375587,,, -11957446,36455,13797,othdiscr,2,False,3,687,784,1494680,social,14.665238994089853,21,SHARED2FREE,0.8916379345159234,,, -11957447,36455,13797,othdiscr,3,False,3,580,687,1494680,home,,21,WALK,1.6583917720273251,,, -11957553,36455,13797,work,1,True,1,562,580,1494694,work,,8,WALK,1.0769131864309256,,, -11957557,36455,13797,work,1,False,1,580,562,1494694,home,,12,SHARED3FREE,1.1126409173456748,,, -13679289,41705,15777,eatout,1,True,1,1070,645,1709911,eatout,,9,WALK_TRANSIT,3.1199845193778537,1500.0,1611.0,fastest -13679293,41705,15777,eatout,1,False,1,645,1070,1709911,home,,15,WALK_TRANSIT,1.5023958487827092,1611.0,1500.0,shortest -13679601,41706,15777,atwork,1,True,1,989,854,1709950,atwork,,12,WALK_TRANSIT,1.744638602675838,1618.0,1748.0,fastest -13679605,41706,15777,atwork,1,False,1,854,989,1709950,work,,12,SHARED2FREE,1.1051431249206285,,, -13679881,41706,15777,work,1,True,1,854,645,1709985,work,,8,SHARED2FREE,0.484767983512682,,, -13679885,41706,15777,work,1,False,2,1080,854,1709985,escort,14.795756318103102,18,SHARED2FREE,0.4510582957043532,,, -13679886,41706,15777,work,2,False,2,645,1080,1709985,home,,19,SHARED2FREE,0.4925057974553318,,, -16411585,50035,18261,eatout,1,True,2,897,757,2051448,shopping,9.84722541528066,16,SHARED3FREE,-0.8214696655451806,,, -16411586,50035,18261,eatout,2,True,2,1070,897,2051448,eatout,,17,WALK,0.1729919081975451,,, -16411589,50035,18261,eatout,1,False,1,757,1070,2051448,home,,18,SHARED3FREE,-1.5064572485047776,,, -16411729,50035,18261,school,1,True,1,919,757,2051466,school,,7,SHARED2FREE,0.9176970231226664,,, -16411733,50035,18261,school,1,False,1,757,919,2051466,home,,13,WALK_TRANSIT,1.0330239915909305,1748.0,1618.0,fastest -16411745,50035,18261,shopping,1,True,1,996,757,2051468,shopping,,19,DRIVEALONEFREE,-0.4672462178894699,,, -16411749,50035,18261,shopping,1,False,1,757,996,2051468,home,,19,DRIVEALONEFREE,-0.4597066008562396,,, -16412033,50036,18261,othmaint,1,True,1,877,757,2051504,othmaint,,7,SHARED2FREE,0.8103017184288349,,, -16412037,50036,18261,othmaint,1,False,1,757,877,2051504,home,,14,SHARED2FREE,0.7939513107930013,,, -16412449,50037,18261,work,1,True,1,1070,757,2051556,work,,7,DRIVEALONEFREE,2.631618309801618,,, -16412453,50037,18261,work,1,False,1,757,1070,2051556,home,,16,WALK_TRANSIT,1.2835436251782917,1651.0,1500.0,fastest -18151113,55338,19758,school,1,True,1,699,794,2268889,school,,7,WALK_TRANSIT,1.353066071452224,1608.0,1652.0,cheapest -18151117,55338,19758,school,1,False,1,794,699,2268889,home,,10,SHARED2FREE,1.118204426327703,,, -18151505,55339,19758,work,1,True,1,762,794,2268938,work,,5,DRIVEALONEFREE,-0.0975185793139585,,, -18151509,55339,19758,work,1,False,1,794,762,2268938,home,,16,DRIVEALONEFREE,-0.0075877722519422,,, -18990529,57897,20552,work,1,True,1,1070,829,2373816,work,,7,WALK_TRANSIT,2.259124587532973,1500.0,1754.0,fastest -18990533,57897,20552,work,1,False,1,829,1070,2373816,home,,20,DRIVEALONEFREE,1.0944715206029072,,, -18990545,57898,20552,atwork,1,True,2,948,948,2373818,othmaint,7.505984009752515,11,WALK,-0.4339456610259058,,, -18990546,57898,20552,atwork,2,True,2,948,948,2373818,atwork,,11,WALK,-0.4339456610259058,,, -18990549,57898,20552,atwork,1,False,1,948,948,2373818,work,,13,WALK,-0.4339456610259058,,, -18990857,57898,20552,work,1,True,1,948,829,2373857,work,,11,DRIVEALONEFREE,-0.4249793219573366,,, -18990861,57898,20552,work,1,False,2,739,948,2373857,escort,11.536722530937924,17,DRIVEALONEFREE,-0.4351552970051177,,, -18990862,57898,20552,work,2,False,2,829,739,2373857,home,,17,DRIVEALONEFREE,-0.0991454272062134,,, -18991185,57899,20552,work,1,True,1,687,829,2373898,work,,7,WALK,0.5220452718494769,,, -18991189,57899,20552,work,1,False,1,829,687,2373898,home,,17,WALK,0.6195549538462863,,, -18991841,57901,20552,work,1,True,1,708,829,2373980,work,,6,DRIVEALONEFREE,-0.2156500818117738,,, -18991845,57901,20552,work,1,False,1,829,708,2373980,home,,16,DRIVEALONEFREE,-0.82405708338336,,, -18991849,57901,20552,work,1,True,2,687,829,2373981,othmaint,13.312507041737812,18,WALK,1.203803918419165,,, -18991850,57901,20552,work,2,True,2,708,687,2373981,work,,21,SHARED2FREE,0.5645998198729414,,, -18991853,57901,20552,work,1,False,1,829,708,2373981,home,,21,DRIVEALONEFREE,0.1538836282621585,,, -20510417,62531,21869,school,1,True,1,938,900,2563802,school,,20,TAXI,1.0846519720828849,,, -20510421,62531,21869,school,1,False,1,900,938,2563802,home,,20,WALK,1.0317475525946904,,, -20510569,62532,21869,escort,1,True,1,647,900,2563821,escort,,6,SHARED2FREE,0.335383128230131,,, -20510573,62532,21869,escort,1,False,1,900,647,2563821,home,,7,DRIVEALONEFREE,0.3214092822815929,,, -20510897,62533,21869,escort,1,True,1,695,900,2563862,escort,,5,SHARED3FREE,0.7146320921116115,,, -20510901,62533,21869,escort,1,False,4,996,695,2563862,shopping,13.9010789936427,6,SHARED3FREE,0.8524136187385734,,, -20510902,62533,21869,escort,2,False,4,565,996,2563862,eatout,14.39637292108163,6,SHARED2FREE,0.5414028775934869,,, -20510903,62533,21869,escort,3,False,4,1099,565,2563862,escort,14.873416692380914,6,SHARED3FREE,0.5359149885298558,,, -20510904,62533,21869,escort,4,False,4,900,1099,2563862,home,,6,SHARED2FREE,0.8757485870243559,,, -20510905,62533,21869,escort,1,True,1,518,900,2563863,escort,,11,SHARED2FREE,0.403734784317824,,, -20510909,62533,21869,escort,1,False,1,900,518,2563863,home,,11,SHARED3FREE,0.4094721213082503,,, -20510913,62533,21869,escort,1,True,1,844,900,2563864,escort,,14,SHARED2FREE,0.2145990810721203,,, -20510917,62533,21869,escort,1,False,1,900,844,2563864,home,,14,SHARED2FREE,0.2175421651558981,,, -20511025,62533,21869,othdiscr,1,True,1,1070,900,2563878,othdiscr,,11,BIKE,3.2327250107654435,,, -20511029,62533,21869,othdiscr,1,False,1,900,1070,2563878,home,,12,BIKE,1.820941282963293,,, -20511401,62534,21869,school,1,True,1,793,900,2563925,school,,8,SHARED3FREE,0.6855068986830097,,, -20511405,62534,21869,school,1,False,1,900,793,2563925,home,,9,SHARED3FREE,0.6540064639192135,,, -22303745,67999,23619,escort,1,True,1,767,973,2787968,escort,,13,DRIVEALONEFREE,0.890462913926255,,, -22303749,67999,23619,escort,1,False,3,1023,767,2787968,eatout,15.21145768299369,13,DRIVEALONEFREE,0.9380038576909148,,, -22303750,67999,23619,escort,2,False,3,993,1023,2787968,escort,16.38782485475755,13,DRIVEALONEFREE,1.1641324346709314,,, -22303751,67999,23619,escort,3,False,3,973,993,2787968,home,,13,SHARED3FREE,1.1940117377263124,,, -22303961,67999,23619,social,1,True,1,988,973,2787995,social,,17,WALK,-0.5648561131494263,,, -22303965,67999,23619,social,1,False,1,973,988,2787995,home,,20,WALK,-0.532826088016878,,, -22304313,68000,23619,work,1,True,2,949,973,2788039,social,22.753813938189783,7,SHARED3FREE,1.149142351882263,,, -22304314,68000,23619,work,2,True,2,1070,949,2788039,work,,8,WALK_TRANSIT,3.309993337941638,1500.0,1747.0,fastest -22304317,68000,23619,work,1,False,1,973,1070,2788039,home,,21,WALK_TRANSIT,1.875153102901004,1588.0,1500.0,fastest -25904705,78977,26897,school,1,True,1,984,1081,3238088,school,,7,WALK,-0.4961040893656598,,, -25904709,78977,26897,school,1,False,1,1081,984,3238088,home,,14,WALK,-0.4863570793073567,,, -25905145,78979,26897,atwork,1,True,1,1070,897,3238143,atwork,,9,DRIVEALONEFREE,-0.0790579016887315,,, -25905149,78979,26897,atwork,1,False,1,897,1070,3238143,work,,11,DRIVEALONEFREE,0.0896991164812165,,, -25905425,78979,26897,work,1,True,1,897,1081,3238178,work,,7,DRIVEALONEFREE,0.0543705077473041,,, -25905429,78979,26897,work,1,False,1,1081,897,3238178,home,,18,DRIVEALONEFREE,0.2237278343269858,,, -421021769,1283602,435012,work,1,True,1,1078,521,52627721,work,,8,WALK_TRANSIT,1.8939124714977291,1584.0,1608.0,fastest -421021773,1283602,435012,work,1,False,1,521,1078,52627721,home,,18,WALK_TRANSIT,0.9194329115009408,1238.0,1584.0,shortest -421108705,1283868,435278,atwork,1,True,1,1070,1070,52638588,atwork,,12,WALK,2.4750620100011926,,, -421108709,1283868,435278,atwork,1,False,1,1070,1070,52638588,work,,12,WALK,2.4750620100011926,,, -421108753,1283868,435278,eatout,1,True,1,1070,537,52638594,eatout,,18,WALK_TRANSIT,3.4974794156920828,1500.0,1558.0,shortest -421108757,1283868,435278,eatout,1,False,2,895,1070,52638594,shopping,17.316225456083114,21,WALK,2.514244617152652,,, -421108758,1283868,435278,eatout,2,False,2,537,895,52638594,home,,21,DRIVEALONEFREE,1.0614628362390803,,, -421109017,1283868,435278,work,1,True,1,1070,537,52638627,work,,9,WALK_TRANSIT,2.6311889287575188,1500.0,1558.0,cheapest -421109021,1283868,435278,work,1,False,1,537,1070,52638627,home,,18,SHARED2FREE,1.283382912854287,,, -421134601,1283946,435356,work,1,True,1,1070,523,52641825,work,,7,WALK_TRANSIT,2.648329293837569,1500.0,1604.0,shortest -421134605,1283946,435356,work,1,False,1,523,1070,52641825,home,,19,TNC_SINGLE,1.3977246630790026,,, -421348457,1284598,436008,work,1,True,1,553,562,52668557,work,,9,WALK,0.3477587678510122,,, -421348461,1284598,436008,work,1,False,1,562,553,52668557,home,,19,WALK,0.3521744962789443,,, -421878553,1286215,437625,atwork,1,True,1,1077,606,52734819,atwork,,10,WALK_TRANSIT,1.186699589189828,1500.0,1562.0,cheapest -421878557,1286215,437625,atwork,1,False,1,606,1077,52734819,work,,13,WALK_TRANSIT,1.681572479856284,1562.0,1500.0,fastest -421878833,1286215,437625,work,1,True,1,606,656,52734854,work,,8,DRIVEALONEFREE,0.1102478757651215,,, -421878837,1286215,437625,work,1,False,3,934,606,52734854,othmaint,11.66765616556947,17,DRIVEALONEFREE,-0.3317318054812608,,, -421878838,1286215,437625,work,2,False,3,730,934,52734854,othmaint,11.283765093022993,19,DRIVEALONEFREE,-0.1643308499157778,,, -421878839,1286215,437625,work,3,False,3,656,730,52734854,home,,19,DRIVEALONEFREE,0.0934059914274113,,, -423180353,1290184,441594,atwork,1,True,1,732,977,52897544,atwork,,11,WALK,2.454959930326644,,, -423180357,1290184,441594,atwork,1,False,1,977,732,52897544,work,,11,SHARED2FREE,2.474151530078552,,, -423180401,1290184,441594,eatout,1,True,1,949,1096,52897550,eatout,,21,SHARED2FREE,0.0292772650529429,,, -423180405,1290184,441594,eatout,1,False,2,1070,949,52897550,othmaint,13.725541816697769,21,DRIVEALONEFREE,-0.5109700761493449,,, -423180406,1290184,441594,eatout,2,False,2,1096,1070,52897550,home,,21,WALK,1.478188695251212,,, -423180553,1290184,441594,othdiscr,1,True,1,684,1096,52897569,othdiscr,,5,DRIVEALONEFREE,-0.0552559508415055,,, -423180557,1290184,441594,othdiscr,1,False,1,1096,684,52897569,home,,7,TAXI,-0.0112939168809909,,, -423180561,1290184,441594,othdiscr,1,True,1,762,1096,52897570,othdiscr,,21,WALK_TRANSIT,1.8308609456730651,1748.0,1516.0,fastest -423180565,1290184,441594,othdiscr,1,False,1,1096,762,52897570,home,,22,DRIVEALONEFREE,1.1096229314740322,,, -423180569,1290184,441594,othdiscr,1,True,1,725,1096,52897571,othdiscr,,23,WALK,-3.2668547335399403,,, -423180573,1290184,441594,othdiscr,1,False,1,1096,725,52897571,home,,23,WALK,-3.2660886555615467,,, -423180665,1290184,441594,work,1,True,1,977,1096,52897583,work,,9,DRIVEALONEFREE,-0.1968641598702008,,, -423180669,1290184,441594,work,1,False,2,996,977,52897583,shopping,10.57412967355008,20,WALK,0.2087211377734316,,, -423180670,1290184,441594,work,2,False,2,1096,996,52897583,home,,21,DRIVEALONEFREE,-0.1682790295025423,,, -423325641,1290626,442036,work,1,True,2,1070,1100,52915705,work,23.18866767080017,8,WALK_TRANSIT,2.8246315365649477,1500.0,1584.0,cheapest -423325642,1290626,442036,work,2,True,2,1070,1070,52915705,work,,9,TAXI,3.132049389519477,,, -423325645,1290626,442036,work,1,False,1,1100,1070,52915705,home,,18,WALK_TRANSIT,1.8829102540483185,1584.0,1500.0,shortest -611033041,1862905,721960,othdiscr,1,True,1,986,960,76379130,othdiscr,,15,WALK_TRANSIT,1.4885049380905109,1618.0,1762.0,shortest -611033045,1862905,721960,othdiscr,1,False,1,960,986,76379130,home,,21,TAXI,1.3532506038209091,,, -611033369,1862906,721960,othdiscr,1,True,1,992,960,76379171,othdiscr,,11,SHARED3FREE,0.8858713264739554,,, -611033373,1862906,721960,othdiscr,1,False,2,551,992,76379171,eatout,14.365988124910029,16,DRIVEALONEFREE,0.6551993708892542,,, -611033374,1862906,721960,othdiscr,2,False,2,960,551,76379171,home,,16,SHARED2FREE,0.5117735347645973,,, -647572569,1974306,760593,othdiscr,1,True,1,564,509,80946571,othdiscr,,8,WALK,0.1247752862685705,,, -647572573,1974306,760593,othdiscr,1,False,1,509,564,80946571,home,,16,WALK,0.116845013634565,,, -647572729,1974307,760593,atwork,1,True,1,913,739,80946591,atwork,,12,SHARED3FREE,2.326963701825421,,, -647572733,1974307,760593,atwork,1,False,1,739,913,80946591,work,,13,SHARED3FREE,2.32846863119764,,, -647573009,1974307,760593,work,1,True,2,741,509,80946626,shopping,18.29690262001317,9,WALK_TRANSIT,1.757205419198924,1663.0,1333.0,fastest -647573010,1974307,760593,work,2,True,2,739,741,80946626,work,,10,BIKE,2.23760901377759,,, -647573013,1974307,760593,work,1,False,2,518,739,80946626,work,19.081960514674996,17,WALK_TRANSIT,2.9100248711462138,1681.0,1662.0,cheapest -647573014,1974307,760593,work,2,False,2,509,518,80946626,home,,18,WALK,2.5433977364719564,,, -647573097,1974308,760593,escort,1,True,1,716,509,80946637,escort,,5,SHARED3FREE,0.8908384736539703,,, -647573101,1974308,760593,escort,1,False,1,509,716,80946637,home,,5,SHARED3FREE,0.8921116412562625,,, -648387521,1976791,761445,escort,1,True,1,908,613,81048440,escort,,13,SHARED2FREE,0.431874605975145,,, -648387525,1976791,761445,escort,1,False,1,613,908,81048440,home,,13,SHARED2FREE,0.4276938639754877,,, -648387809,1976792,761445,atwork,1,True,1,517,517,81048476,atwork,,9,WALK,3.011532825794776,,, -648387813,1976792,761445,atwork,1,False,1,517,517,81048476,work,,9,WALK,3.011532825794776,,, -648387825,1976792,761445,eatout,1,True,1,648,613,81048478,eatout,,13,DRIVEALONEFREE,0.4724646811902892,,, -648387829,1976792,761445,eatout,1,False,1,613,648,81048478,home,,19,TAXI,0.5571365851477499,,, -648388089,1976792,761445,work,1,True,1,517,613,81048511,work,,5,WALK_TRANSIT,1.796194897875614,1681.0,1238.0,cheapest -648388093,1976792,761445,work,1,False,1,613,517,81048511,home,,12,DRIVEALONEFREE,1.474443497861856,,, -649042753,1978788,762159,social,1,True,1,763,961,81130344,social,,8,SHARED2FREE,0.3356357191681573,,, -649042757,1978788,762159,social,1,False,1,961,763,81130344,home,,20,SHARED2FREE,0.3505516593935055,,, -649043193,1978790,762159,escort,1,True,1,992,961,81130399,escort,,8,SHARED2FREE,0.5305214939922914,,, -649043197,1978790,762159,escort,1,False,1,961,992,81130399,home,,8,SHARED2FREE,0.5188517472186277,,, -649043433,1978790,762159,work,1,True,1,672,961,81130429,work,,8,DRIVEALONEFREE,-0.3341036694423639,,, -649043437,1978790,762159,work,1,False,1,961,672,81130429,home,,17,DRIVEALONEFREE,-0.3318966430958522,,, -649043761,1978791,762159,work,1,True,1,1070,961,81130470,work,,7,WALK_TRANSIT,2.6922733054492127,1500.0,1762.0,fastest -649043765,1978791,762159,work,1,False,1,961,1070,81130470,home,,17,TAXI,1.4645945681666606,,, -819359665,2498047,922602,school,1,True,1,865,730,102419958,school,,9,TAXI,1.202896883865351,,, -819359669,2498047,922602,school,1,False,1,730,865,102419958,home,,16,SHARED2FREE,1.846403103708924,,, -819360057,2498048,922602,work,1,True,1,735,730,102420007,work,,7,SHARED3FREE,1.7808819076757447,,, -819360061,2498048,922602,work,1,False,1,730,735,102420007,home,,16,WALK_TRANSIT,2.022371451590549,1612.0,1592.0,shortest -819360385,2498049,922602,work,1,True,1,763,730,102420048,work,,6,WALK,-0.7841986127882443,,, -819360389,2498049,922602,work,1,False,1,730,763,102420048,home,,16,WALK,-0.8403247829423018,,, -860079225,2622192,952720,school,1,True,1,995,1025,107509903,school,,7,TNC_SINGLE,1.737007403541493,,, -860079229,2622192,952720,school,1,False,1,1025,995,107509903,home,,14,SHARED3FREE,1.6399355775465674,,, -860079377,2622193,952720,escort,1,True,1,773,1025,107509922,escort,,6,SHARED3FREE,0.9453815876415104,,, -860079381,2622193,952720,escort,1,False,1,1025,773,107509922,home,,6,SHARED2FREE,0.9690872179986344,,, -860079529,2622193,952720,othmaint,1,True,1,550,1025,107509941,othmaint,,8,TAXI,1.6791134568237254,,, -860079533,2622193,952720,othmaint,1,False,1,1025,550,107509941,home,,15,WALK_TRANSIT,1.6848126498987464,1666.0,1238.0,shortest -860079897,2622194,952720,shopping,1,True,1,989,1025,107509987,shopping,,9,WALK,1.6280881165461685,,, -860079901,2622194,952720,shopping,1,False,1,1025,989,107509987,home,,11,SHARED3FREE,1.5894514263370938,,, -860080273,2622195,952720,work,1,True,1,1021,1025,107510034,work,,8,DRIVEALONEFREE,0.1307608725915786,,, -860080277,2622195,952720,work,1,False,1,1025,1021,107510034,home,,17,DRIVEALONEFREE,0.1511979688414838,,, -933123249,2844887,1028031,work,1,True,1,845,846,116640406,work,,11,SHARED3FREE,1.662725895784853,,, -933123253,2844887,1028031,work,1,False,1,846,845,116640406,home,,23,DRIVEALONEFREE,1.6190252500331577,,, -962301409,2933845,1048898,school,1,True,1,666,574,120287676,school,,12,WALK,-1.1062441750768537,,, -962301413,2933845,1048898,school,1,False,1,574,666,120287676,home,,21,WALK,-1.227339677708164,,, -962301737,2933846,1048898,school,1,True,1,515,574,120287717,school,,8,SHARED2FREE,-2.095040038174043,,, -962301741,2933846,1048898,school,1,False,1,574,515,120287717,home,,16,SHARED2FREE,-2.102822435414125,,, -962302017,2933847,1048898,othdiscr,1,True,1,657,574,120287752,othdiscr,,7,WALK_TRANSIT,1.5362116862704658,1238.0,1559.0,cheapest -962302021,2933847,1048898,othdiscr,1,False,2,1070,657,120287752,eatout,17.04052424640584,12,WALK_TRANSIT,3.494101406816589,3.0,1238.0,shortest -962302022,2933847,1048898,othdiscr,2,False,2,574,1070,120287752,home,,12,WALK_TRANSIT,1.3230638799617858,1559.0,1500.0,fastest -962302457,2933848,1048898,work,1,True,1,502,574,120287807,work,,6,BIKE,1.3439545881637307,,, -962302461,2933848,1048898,work,1,False,1,574,502,120287807,home,,18,WALK_TRANSIT,1.3169033457581467,1559.0,1333.0,shortest -1055052265,3216622,1148260,univ,1,True,1,1074,1076,131881533,univ,,14,TAXI,0.933892780827025,,, -1055052269,3216622,1148260,univ,1,False,1,1076,1074,131881533,home,,15,BIKE,0.8615934658413547,,, +10872201,33146,12593,work,1,True,3,350,68,1359025,escort,11.62710514749821,5,DRIVEALONEFREE,-0.3153159028910412,,, +10872202,33146,12593,work,2,True,3,491,350,1359025,escort,12.424454211009824,7,DRIVEALONEFREE,-0.1330752275781039,,, +10872203,33146,12593,work,3,True,3,608,491,1359025,work,,7,DRIVEALONEFREE,0.1534821760951137,,, +10872205,33146,12593,work,1,False,1,68,608,1359025,home,,17,WALK,-0.3102996602636911,,, +10872529,33147,12593,work,1,True,1,578,68,1359066,work,,8,WALK_TRANSIT,2.7167630046939766,1500.0,1558.0,shortest +10872533,33147,12593,work,1,False,2,578,578,1359066,escort,19.372183137979995,15,WALK,2.9941670645943352,,, +10872534,33147,12593,work,2,False,2,68,578,1359066,home,,15,WALK,1.93207085094047,,, +11957177,36454,13797,shopping,1,True,1,233,88,1494647,shopping,,13,DRIVEALONEFREE,0.1611535351527045,,, +11957181,36454,13797,shopping,1,False,2,423,233,1494647,shopping,12.025184457266546,17,TAXI,-0.0282876166283608,,, +11957182,36454,13797,shopping,2,False,2,88,423,1494647,home,,17,DRIVEALONEFREE,0.0612368483543265,,, +11957185,36454,13797,shopping,1,True,2,195,88,1494648,escort,13.40305488007496,18,WALK,0.3896474852723683,,, +11957186,36454,13797,shopping,2,True,2,237,195,1494648,shopping,,18,DRIVEALONEFREE,0.4918839198395139,,, +11957189,36454,13797,shopping,1,False,1,88,237,1494648,home,,18,TAXI,0.2042092214903569,,, +11957441,36455,13797,othdiscr,1,True,1,131,88,1494680,othdiscr,,16,DRIVEALONEFREE,0.1718894334261086,,, +11957445,36455,13797,othdiscr,1,False,3,578,131,1494680,othmaint,12.886372901931878,21,TAXI,-0.1803784427916115,,, +11957446,36455,13797,othdiscr,2,False,3,578,578,1494680,social,14.222827339077435,21,WALK,1.1995504881185008,,, +11957447,36455,13797,othdiscr,3,False,3,88,578,1494680,home,,21,TAXI,0.0855272375513731,,, +11957553,36455,13797,work,1,True,1,70,88,1494694,work,,8,WALK,0.2791473443211477,,, +11957557,36455,13797,work,1,False,1,88,70,1494694,home,,12,WALK,0.2791984027393967,,, +13679289,41705,15777,eatout,1,True,1,578,153,1709911,eatout,,9,WALK_TRANSIT,3.0405322129710664,1500.0,1611.0,fastest +13679293,41705,15777,eatout,1,False,1,153,578,1709911,home,,15,TAXI,1.1068614998173167,,, +13679601,41706,15777,atwork,1,True,1,497,362,1709950,atwork,,12,WALK_TRANSIT,1.7190241715037766,1618.0,1748.0,fastest +13679605,41706,15777,atwork,1,False,1,362,497,1709950,work,,12,SHARED2FREE,1.0527994338149784,,, +13679881,41706,15777,work,1,True,1,362,153,1709985,work,,8,SHARED2FREE,0.5758380783277821,,, +13679885,41706,15777,work,1,False,2,597,362,1709985,escort,14.841057893338451,18,SHARED2FREE,0.4353503862435287,,, +13679886,41706,15777,work,2,False,2,153,597,1709985,home,,19,SHARED2FREE,0.4996880766938729,,, +16411585,50035,18261,eatout,1,True,2,498,265,2051448,shopping,10.05333950335537,16,SHARED3FREE,-0.6138900950914048,,, +16411586,50035,18261,eatout,2,True,2,578,498,2051448,eatout,,17,WALK,-0.1273881390654513,,, +16411589,50035,18261,eatout,1,False,1,265,578,2051448,home,,18,SHARED3FREE,-1.5064572485047776,,, +16411729,50035,18261,school,1,True,1,427,265,2051466,school,,7,SHARED3FREE,0.7485359100139714,,, +16411733,50035,18261,school,1,False,1,265,427,2051466,home,,13,WALK_TRANSIT,0.8841733805272283,1748.0,1618.0,fastest +16411745,50035,18261,shopping,1,True,1,504,265,2051468,shopping,,19,DRIVEALONEFREE,-0.4672462178894699,,, +16411749,50035,18261,shopping,1,False,1,265,504,2051468,home,,19,DRIVEALONEFREE,-0.4597066008562396,,, +16412033,50036,18261,othmaint,1,True,1,385,265,2051504,othmaint,,7,SHARED2FREE,0.8019298221360518,,, +16412037,50036,18261,othmaint,1,False,1,265,385,2051504,home,,14,SHARED2FREE,0.7815669264302088,,, +16412449,50037,18261,work,1,True,1,578,265,2051556,work,,7,SHARED2FREE,2.569476979448462,,, +16412453,50037,18261,work,1,False,1,265,578,2051556,home,,16,WALK_TRANSIT,1.0586461337073116,1651.0,1500.0,fastest +18151113,55338,19758,school,1,True,1,207,302,2268889,school,,7,WALK_TRANSIT,1.3220566998920782,1608.0,1652.0,cheapest +18151117,55338,19758,school,1,False,1,302,207,2268889,home,,10,SHARED2FREE,1.0450588497025504,,, +18151505,55339,19758,work,1,True,1,254,302,2268938,work,,5,DRIVEALONEFREE,0.1112722333894017,,, +18151509,55339,19758,work,1,False,1,302,254,2268938,home,,16,DRIVEALONEFREE,0.1112770043068081,,, +18990529,57897,20552,work,1,True,1,578,337,2373816,work,,7,WALK_TRANSIT,2.167693637863771,1500.0,1754.0,fastest +18990533,57897,20552,work,1,False,1,337,578,2373816,home,,20,SHARED2FREE,0.8134213565684749,,, +18990545,57898,20552,atwork,1,True,2,456,456,2373818,othmaint,6.956302743667657,11,WALK,-0.5725554350319478,,, +18990546,57898,20552,atwork,2,True,2,456,456,2373818,atwork,,11,WALK,-0.5725554350319478,,, +18990549,57898,20552,atwork,1,False,1,456,456,2373818,work,,13,WALK,-0.5725554350319478,,, +18990857,57898,20552,work,1,True,1,456,337,2373857,work,,11,DRIVEALONEFREE,-0.4249793219573366,,, +18990861,57898,20552,work,1,False,2,350,456,2373857,escort,11.522885117323073,17,DRIVEALONEFREE,-0.4224341222895262,,, +18990862,57898,20552,work,2,False,2,337,350,2373857,home,,17,DRIVEALONEFREE,-0.0678724579075491,,, +18991185,57899,20552,work,1,True,1,195,337,2373898,work,,7,WALK,0.497663225159644,,, +18991189,57899,20552,work,1,False,1,337,195,2373898,home,,17,WALK,0.4231764292042089,,, +18991841,57901,20552,work,1,True,1,216,337,2373980,work,,6,DRIVEALONEFREE,-0.2156500818117738,,, +18991845,57901,20552,work,1,False,1,337,216,2373980,home,,17,DRIVEALONEFREE,-0.8860210329051176,,, +18991849,57901,20552,work,1,True,2,195,337,2373981,othmaint,13.258114756000287,19,WALK,1.189890260596198,,, +18991850,57901,20552,work,2,True,2,216,195,2373981,work,,19,SHARED2FREE,0.5495070616058875,,, +18991853,57901,20552,work,1,False,1,337,216,2373981,home,,19,DRIVEALONEFREE,0.2682235453386703,,, +20510417,62531,21869,school,1,True,1,442,408,2563802,school,,20,TAXI,0.9918167280007733,,, +20510421,62531,21869,school,1,False,1,408,442,2563802,home,,20,SHARED3FREE,0.9273027824432368,,, +20510569,62532,21869,escort,1,True,1,155,408,2563821,escort,,6,SHARED2FREE,0.335383128230131,,, +20510573,62532,21869,escort,1,False,1,408,155,2563821,home,,7,DRIVEALONEFREE,0.3214092822815929,,, +20510897,62533,21869,escort,1,True,1,203,408,2563862,escort,,5,SHARED3FREE,0.7146320921116115,,, +20510901,62533,21869,escort,1,False,4,504,203,2563862,shopping,13.905386682389064,6,SHARED3FREE,0.8499729610956421,,, +20510902,62533,21869,escort,2,False,4,73,504,2563862,eatout,14.52262097584395,6,SHARED2FREE,0.5819032445094066,,, +20510903,62533,21869,escort,3,False,4,607,73,2563862,escort,14.887504187131842,6,SHARED3FREE,0.5359149885298558,,, +20510904,62533,21869,escort,4,False,4,408,607,2563862,home,,6,SHARED2FREE,0.8747878537896981,,, +20510905,62533,21869,escort,1,True,1,26,408,2563863,escort,,11,SHARED2FREE,0.403734784317824,,, +20510909,62533,21869,escort,1,False,1,408,26,2563863,home,,11,SHARED3FREE,0.4094721213082503,,, +20510913,62533,21869,escort,1,True,1,352,408,2563864,escort,,14,SHARED2FREE,0.2145990810721203,,, +20510917,62533,21869,escort,1,False,1,408,352,2563864,home,,14,SHARED2FREE,0.2175421651558981,,, +20511025,62533,21869,othdiscr,1,True,1,578,408,2563878,othdiscr,,11,WALK_TRANSIT,3.124262835713407,1500.0,73.0,fastest +20511029,62533,21869,othdiscr,1,False,1,408,578,2563878,home,,12,WALK_TRANSIT,1.320541626578869,73.0,1500.0,fastest +20511401,62534,21869,school,1,True,1,301,408,2563925,school,,8,SHARED3FREE,0.6297085882553292,,, +20511405,62534,21869,school,1,False,1,408,301,2563925,home,,9,SHARED3FREE,0.5963697573192646,,, +22303745,67999,23619,escort,1,True,1,269,481,2787968,escort,,13,DRIVEALONEFREE,0.9471497272884896,,, +22303749,67999,23619,escort,1,False,3,514,269,2787968,eatout,15.292255065772112,13,DRIVEALONEFREE,1.0997559458267394,,, +22303750,67999,23619,escort,2,False,3,496,514,2787968,escort,16.470852149311927,13,DRIVEALONEFREE,0.9919349481465494,,, +22303751,67999,23619,escort,3,False,3,481,496,2787968,home,,13,SHARED2FREE,1.0470193699425838,,, +22303961,67999,23619,social,1,True,1,496,481,2787995,social,,17,WALK,-0.5194290001855648,,, +22303965,67999,23619,social,1,False,1,481,496,2787995,home,,20,WALK,-0.5194329974199648,,, +22304313,68000,23619,work,1,True,2,437,481,2788039,social,22.336188858055703,7,SHARED3FREE,1.245370428774317,,, +22304314,68000,23619,work,2,True,2,578,437,2788039,work,,8,WALK_TRANSIT,3.2620308996092504,1500.0,1717.0,fastest +22304317,68000,23619,work,1,False,1,481,578,2788039,home,,21,WALK_TRANSIT,1.5243469382674963,1588.0,1500.0,fastest +25904705,78977,26897,school,1,True,1,492,589,3238088,school,,7,WALK,1.1991023619246646,,, +25904709,78977,26897,school,1,False,1,589,492,3238088,home,,14,WALK_TRANSIT,1.1320593632124774,1584.0,1618.0,cheapest +25905145,78979,26897,atwork,1,True,1,578,405,3238143,atwork,,9,DRIVEALONEFREE,-0.0496509953086405,,, +25905149,78979,26897,atwork,1,False,1,405,578,3238143,work,,11,DRIVEALONEFREE,-0.0461032274958093,,, +25905425,78979,26897,work,1,True,1,405,589,3238178,work,,7,DRIVEALONEFREE,0.119565485964625,,, +25905429,78979,26897,work,1,False,1,589,405,3238178,home,,18,DRIVEALONEFREE,0.1179742410301269,,, +421021769,1283602,435012,work,1,True,1,586,29,52627721,work,,8,WALK_TRANSIT,1.8021163156603703,1584.0,1608.0,fastest +421021773,1283602,435012,work,1,False,1,29,586,52627721,home,,18,WALK_TRANSIT,0.8450040063014843,1238.0,1584.0,shortest +421108705,1283868,435278,atwork,1,True,1,578,578,52638588,atwork,,12,WALK,2.459897236353819,,, +421108709,1283868,435278,atwork,1,False,1,578,578,52638588,work,,12,WALK,2.459897236353819,,, +421108753,1283868,435278,eatout,1,True,1,578,45,52638594,eatout,,18,WALK_TRANSIT,3.4599905231494112,1500.0,1558.0,shortest +421108757,1283868,435278,eatout,1,False,2,531,578,52638594,shopping,16.971131751329413,21,WALK,2.4125169566233917,,, +421108758,1283868,435278,eatout,2,False,2,45,531,52638594,home,,21,SHARED2FREE,0.9951467720082716,,, +421109017,1283868,435278,work,1,True,1,578,45,52638627,work,,9,WALK_TRANSIT,2.556266300862197,1500.0,1558.0,cheapest +421109021,1283868,435278,work,1,False,1,45,578,52638627,home,,18,WALK,1.4769191108284825,,, +421134601,1283946,435356,work,1,True,1,578,31,52641825,work,,7,WALK_TRANSIT,2.564484302870523,1500.0,1604.0,shortest +421134605,1283946,435356,work,1,False,1,31,578,52641825,home,,19,TNC_SINGLE,1.2220841443471149,,, +421348457,1284598,436008,work,1,True,1,61,70,52668557,work,,9,WALK,0.38649989959854,,, +421348461,1284598,436008,work,1,False,1,70,61,52668557,home,,19,WALK,0.3859946435001655,,, +421878553,1286215,437625,atwork,1,True,1,585,114,52734819,atwork,,10,WALK_TRANSIT,1.081956268867972,1500.0,1562.0,cheapest +421878557,1286215,437625,atwork,1,False,1,114,585,52734819,work,,13,WALK_TRANSIT,1.6218413932594848,1562.0,1500.0,fastest +421878833,1286215,437625,work,1,True,1,114,164,52734854,work,,8,DRIVEALONEFREE,0.0108507049503425,,, +421878837,1286215,437625,work,1,False,3,505,114,52734854,othmaint,11.678285931399188,17,DRIVEALONEFREE,-0.2251500347921951,,, +421878838,1286215,437625,work,2,False,3,238,505,52734854,othmaint,11.653319784580876,19,DRIVEALONEFREE,-0.0918226143933905,,, +421878839,1286215,437625,work,3,False,3,164,238,52734854,home,,19,DRIVEALONEFREE,0.1142972915542419,,, +423180353,1290184,441594,atwork,1,True,1,240,485,52897544,atwork,,11,WALK,1.1616704858421698,,, +423180357,1290184,441594,atwork,1,False,1,485,240,52897544,work,,11,WALK,1.1616704548482053,,, +423180401,1290184,441594,eatout,1,True,1,347,604,52897550,eatout,,21,SHARED2FREE,0.1039931558703346,,, +423180405,1290184,441594,eatout,1,False,2,600,347,52897550,othmaint,13.35678663215775,21,DRIVEALONEFREE,0.1168155383854849,,, +423180406,1290184,441594,eatout,2,False,2,604,600,52897550,home,,21,WALK,0.9240213001163314,,, +423180553,1290184,441594,othdiscr,1,True,1,22,604,52897569,othdiscr,,5,WALK,-0.0748834380443437,,, +423180557,1290184,441594,othdiscr,1,False,1,604,22,52897569,home,,7,TAXI,-0.2702437909392063,,, +423180561,1290184,441594,othdiscr,1,True,1,241,604,52897570,othdiscr,,21,BIKE,-1.0261281159570592,,, +423180565,1290184,441594,othdiscr,1,False,1,604,241,52897570,home,,22,BIKE,-1.0364857021941367,,, +423180569,1290184,441594,othdiscr,1,True,1,161,604,52897571,othdiscr,,23,TAXI,0.93738966972432,,, +423180573,1290184,441594,othdiscr,1,False,2,232,161,52897571,eatout,15.99440404367919,23,SHARED2FREE,1.631333771028581,,, +423180574,1290184,441594,othdiscr,2,False,2,604,232,52897571,home,,23,WALK_TRANSIT,0.5700662414192166,1516.0,1608.0,shortest +423180665,1290184,441594,work,1,True,1,485,604,52897583,work,,9,DRIVEALONEFREE,-0.2074790120685403,,, +423180669,1290184,441594,work,1,False,2,504,485,52897583,shopping,10.648841698636746,20,WALK,0.2135574369121218,,, +423180670,1290184,441594,work,2,False,2,604,504,52897583,home,,21,DRIVEALONEFREE,-0.1688195503141319,,, +423325641,1290626,442036,work,1,True,2,578,608,52915705,work,22.793846679159277,8,WALK_TRANSIT,2.6947489444098363,1500.0,1584.0,cheapest +423325642,1290626,442036,work,2,True,2,578,578,52915705,work,,9,TAXI,2.9785405530718863,,, +423325645,1290626,442036,work,1,False,1,608,578,52915705,home,,18,WALK_TRANSIT,1.5204714046677252,1584.0,1500.0,shortest +611033041,1862905,721960,othdiscr,1,True,1,494,468,76379130,othdiscr,,15,BIKE,-0.0551141326390255,,, +611033045,1862905,721960,othdiscr,1,False,1,468,494,76379130,home,,21,BIKE,-0.0503449080980652,,, +611033369,1862906,721960,othdiscr,1,True,1,500,468,76379171,othdiscr,,11,WALK,-1.4043276286248658,,, +611033373,1862906,721960,othdiscr,1,False,1,468,500,76379171,home,,16,WALK,-1.3842194916923618,,, +647572569,1974306,760593,othdiscr,1,True,1,72,17,80946571,othdiscr,,8,WALK,0.0729983438076541,,, +647572573,1974306,760593,othdiscr,1,False,1,17,72,80946571,home,,16,WALK,0.0730661118420978,,, +647572729,1974307,760593,atwork,1,True,1,324,204,80946591,atwork,,12,SHARED3FREE,2.153561514861549,,, +647572733,1974307,760593,atwork,1,False,1,204,324,80946591,work,,13,SHARED3FREE,2.1604129937838445,,, +647573009,1974307,760593,work,1,True,2,244,17,80946626,shopping,16.850702372805465,9,WALK_TRANSIT,1.2419694217829829,1748.0,1603.0,shortest +647573010,1974307,760593,work,2,True,2,204,244,80946626,work,,10,WALK_TRANSIT,1.6771107500758995,1748.0,1660.0,shortest +647573013,1974307,760593,work,1,False,2,176,204,80946626,work,18.744021387869157,17,WALK_TRANSIT,1.7298767083660849,1333.0,1658.0,shortest +647573014,1974307,760593,work,2,False,2,17,176,80946626,home,,18,SHARED3FREE,1.5876337419644198,,, +647573097,1974308,760593,escort,1,True,1,224,17,80946637,escort,,5,SHARED3FREE,0.8922078793578596,,, +647573101,1974308,760593,escort,1,False,1,17,224,80946637,home,,5,SHARED3FREE,0.8942651098665495,,, +648387521,1976791,761445,escort,1,True,1,483,121,81048440,escort,,13,SHARED2FREE,0.5482474086094538,,, +648387525,1976791,761445,escort,1,False,1,121,483,81048440,home,,13,SHARED2FREE,0.5356122283077811,,, +648388065,1976792,761445,social,1,True,1,494,121,81048508,social,,14,DRIVEALONEFREE,0.4863963073134856,,, +648388069,1976792,761445,social,1,False,1,121,494,81048508,home,,19,TAXI,0.5958102526909501,,, +648388089,1976792,761445,work,1,True,1,25,121,81048511,work,,5,TAXI,1.5879637864872944,,, +648388093,1976792,761445,work,1,False,1,121,25,81048511,home,,12,SHARED2FREE,1.1814737849875496,,, +649042753,1978788,762159,social,1,True,1,271,469,81130344,social,,8,SHARED2FREE,0.3356357191681573,,, +649042757,1978788,762159,social,1,False,1,469,271,81130344,home,,20,SHARED2FREE,0.3505516593935055,,, +649043193,1978790,762159,escort,1,True,1,500,469,81130399,escort,,8,SHARED2FREE,0.5471590891799881,,, +649043197,1978790,762159,escort,1,False,1,469,500,81130399,home,,8,SHARED2FREE,0.5333524059928323,,, +649043433,1978790,762159,work,1,True,1,180,469,81130429,work,,8,DRIVEALONEFREE,-0.3341036694423639,,, +649043437,1978790,762159,work,1,False,1,469,180,81130429,home,,17,DRIVEALONEFREE,-0.3318966430958522,,, +649043761,1978791,762159,work,1,True,1,578,469,81130470,work,,7,WALK_TRANSIT,2.58470148067105,1500.0,1762.0,fastest +649043765,1978791,762159,work,1,False,1,469,578,81130470,home,,17,TAXI,1.0619367021498107,,, +819359665,2498047,922602,school,1,True,1,432,238,102419958,school,,9,TAXI,1.009935909154772,,, +819359669,2498047,922602,school,1,False,1,238,432,102419958,home,,16,SHARED2FREE,1.7812941984342985,,, +819360057,2498048,922602,work,1,True,1,243,238,102420007,work,,7,SHARED3FREE,1.630772738193186,,, +819360061,2498048,922602,work,1,False,1,238,243,102420007,home,,16,WALK_TRANSIT,1.9203484754329128,1612.0,1592.0,shortest +819360385,2498049,922602,work,1,True,1,271,238,102420048,work,,6,WALK,-1.149581787441591,,, +819360389,2498049,922602,work,1,False,1,238,271,102420048,home,,16,WALK,-1.1497709160537464,,, +860079225,2622192,952720,school,1,True,1,501,533,107509903,school,,7,TNC_SINGLE,1.6140702309701105,,, +860079229,2622192,952720,school,1,False,1,533,501,107509903,home,,14,SHARED3FREE,1.5475600453518767,,, +860079377,2622193,952720,escort,1,True,1,281,533,107509922,escort,,6,WALK,-0.5578556373070639,,, +860079381,2622193,952720,escort,1,False,1,533,281,107509922,home,,6,WALK,-0.5573194757596159,,, +860079529,2622193,952720,othmaint,1,True,1,58,533,107509941,othmaint,,8,TAXI,1.632612282318792,,, +860079533,2622193,952720,othmaint,1,False,1,533,58,107509941,home,,15,WALK_TRANSIT,1.504929950475204,1666.0,1238.0,shortest +860079897,2622194,952720,shopping,1,True,1,504,533,107509987,shopping,,9,SHARED3FREE,1.5569490328652331,,, +860079901,2622194,952720,shopping,1,False,1,533,504,107509987,home,,13,SHARED3FREE,1.5615906994764628,,, +860080273,2622195,952720,work,1,True,1,578,533,107510034,work,,8,WALK_TRANSIT,2.753496136291756,1500.0,1666.0,fastest +860080277,2622195,952720,work,1,False,1,533,578,107510034,home,,17,WALK,1.6442514073833965,,, +933123249,2844887,1028031,work,1,True,1,385,354,116640406,work,,11,SHARED3FREE,1.076717359011956,,, +933123253,2844887,1028031,work,1,False,1,354,385,116640406,home,,20,SHARED2FREE,1.076810290051006,,, +962301409,2933845,1048898,school,1,True,1,197,82,120287676,school,,12,WALK_TRANSIT,1.28199080402723,1608.0,1559.0,fastest +962301413,2933845,1048898,school,1,False,1,82,197,120287676,home,,21,WALK_TRANSIT,1.089904965843972,1559.0,1608.0,cheapest +962301737,2933846,1048898,school,1,True,1,23,82,120287717,school,,8,SHARED2FREE,-2.124600564910897,,, +962301741,2933846,1048898,school,1,False,1,82,23,120287717,home,,16,SHARED2FREE,-2.1298532205579024,,, +962302017,2933847,1048898,othdiscr,1,True,1,165,82,120287752,othdiscr,,7,WALK_TRANSIT,1.3340675832011153,1238.0,1559.0,cheapest +962302021,2933847,1048898,othdiscr,1,False,2,578,165,120287752,eatout,16.449525883467185,12,WALK_TRANSIT,3.482863279857927,3.0,1238.0,shortest +962302022,2933847,1048898,othdiscr,2,False,2,82,578,120287752,home,,12,WALK_TRANSIT,1.1970215357429652,1559.0,1500.0,fastest +962302457,2933848,1048898,work,1,True,1,10,82,120287807,work,,6,WALK_TRANSIT,1.075493937379867,1333.0,1559.0,fastest +962302461,2933848,1048898,work,1,False,1,82,10,120287807,home,,18,TAXI,1.0317861580544103,,, +1055052265,3216622,1148260,univ,1,True,1,582,584,131881533,univ,,14,TAXI,0.8904812765761871,,, +1055052269,3216622,1148260,univ,1,False,1,584,582,131881533,home,,15,WALK,0.8480171082358124,,, diff --git a/activitysim/examples/placeholder_sandag/test/regress/final_3_zone_trips_sh.csv b/activitysim/examples/placeholder_sandag/test/regress/final_3_zone_trips_sh.csv new file mode 100644 index 0000000000..c783d9a90f --- /dev/null +++ b/activitysim/examples/placeholder_sandag/test/regress/final_3_zone_trips_sh.csv @@ -0,0 +1,185 @@ +trip_id,person_id,household_id,primary_purpose,trip_num,outbound,trip_count,destination,origin,tour_id,purpose,destination_logsum,depart,trip_mode,mode_choice_logsum,atap,btap,path_set +10872201,33146,12593,work,1,True,3,879,560,1359025,escort,11.70885284520791,5,DRIVEALONEFREE,-0.35487465777330723,,, +10872202,33146,12593,work,2,True,3,989,879,1359025,escort,12.022433281644071,7,DRIVEALONEFREE,-0.361684345054043,,, +10872203,33146,12593,work,3,True,3,1100,989,1359025,work,,7,DRIVEALONEFREE,0.1801428827020189,,, +10872205,33146,12593,work,1,False,1,560,1100,1359025,home,,17,DRIVEALONEFREE,-0.368339088809198,,, +10872529,33147,12593,work,1,True,1,1070,560,1359066,work,,8,WALK_TRANSIT,2.7167630046939766,1500.0,1558.0,shortest +10872533,33147,12593,work,1,False,2,909,1070,1359066,escort,18.768693620226532,15,SHARED2FREE,1.3303021092897034,,, +10872534,33147,12593,work,2,False,2,560,909,1359066,home,,15,SHARED2FREE,1.1209924956902448,,, +11957177,36454,13797,shopping,1,True,1,725,580,1494647,shopping,,13,DRIVEALONEFREE,0.17174489954805924,,, +11957181,36454,13797,shopping,1,False,2,803,725,1494647,shopping,12.016029658018422,17,TAXI,0.03522677284406563,,, +11957182,36454,13797,shopping,2,False,2,580,803,1494647,home,,17,DRIVEALONEFREE,0.07359791064631006,,, +11957185,36454,13797,shopping,1,True,2,687,580,1494648,escort,13.593599471819292,18,WALK,0.7788942047635835,,, +11957186,36454,13797,shopping,2,True,2,729,687,1494648,shopping,,18,DRIVEALONEFREE,0.48921017128166105,,, +11957189,36454,13797,shopping,1,False,1,580,729,1494648,home,,18,TAXI,0.23130769643011947,,, +11957441,36455,13797,othdiscr,1,True,1,623,580,1494680,othdiscr,,16,DRIVEALONEFREE,0.7744647479734814,,, +11957445,36455,13797,othdiscr,1,False,3,784,623,1494680,othmaint,15.224351954248833,21,SHARED2FREE,0.6723499661375587,,, +11957446,36455,13797,othdiscr,2,False,3,687,784,1494680,social,14.665238994089853,21,SHARED2FREE,0.8916379345159234,,, +11957447,36455,13797,othdiscr,3,False,3,580,687,1494680,home,,21,WALK,1.6583917720273251,,, +11957553,36455,13797,work,1,True,1,562,580,1494694,work,,8,WALK,1.0769131864309256,,, +11957557,36455,13797,work,1,False,1,580,562,1494694,home,,12,SHARED3FREE,1.1126409173456748,,, +13679289,41705,15777,eatout,1,True,1,1070,645,1709911,eatout,,9,WALK_TRANSIT,3.0405322129710664,1500.0,1611.0,fastest +13679293,41705,15777,eatout,1,False,1,645,1070,1709911,home,,15,TAXI,1.0589757414053866,,, +13679601,41706,15777,atwork,1,True,1,989,854,1709950,atwork,,12,WALK_TRANSIT,1.744638602675838,1618.0,1748.0,fastest +13679605,41706,15777,atwork,1,False,1,854,989,1709950,work,,12,SHARED2FREE,1.1051431249206285,,, +13679881,41706,15777,work,1,True,1,854,645,1709985,work,,8,SHARED2FREE,0.484767983512682,,, +13679885,41706,15777,work,1,False,2,1080,854,1709985,escort,14.795756318103102,18,SHARED2FREE,0.4510582957043532,,, +13679886,41706,15777,work,2,False,2,645,1080,1709985,home,,19,SHARED2FREE,0.4925057974553318,,, +16411585,50035,18261,eatout,1,True,2,897,757,2051448,shopping,9.847225415280661,16,SHARED3FREE,-0.8214696655451806,,, +16411586,50035,18261,eatout,2,True,2,1070,897,2051448,eatout,,17,WALK,0.17299190819754515,,, +16411589,50035,18261,eatout,1,False,1,757,1070,2051448,home,,18,SHARED3FREE,-1.5064572485047776,,, +16411729,50035,18261,school,1,True,1,919,757,2051466,school,,7,SHARED3FREE,0.7485359100139714,,, +16411733,50035,18261,school,1,False,1,757,919,2051466,home,,13,WALK_TRANSIT,0.8841733805272283,1748.0,1618.0,fastest +16411745,50035,18261,shopping,1,True,1,996,757,2051468,shopping,,19,DRIVEALONEFREE,-0.46724621788946996,,, +16411749,50035,18261,shopping,1,False,1,757,996,2051468,home,,19,DRIVEALONEFREE,-0.45970660085623966,,, +16412033,50036,18261,othmaint,1,True,1,877,757,2051504,othmaint,,7,SHARED2FREE,0.8103017184288349,,, +16412037,50036,18261,othmaint,1,False,1,757,877,2051504,home,,14,SHARED2FREE,0.7939513107930013,,, +16412449,50037,18261,work,1,True,1,1070,757,2051556,work,,7,SHARED2FREE,2.569476979448462,,, +16412453,50037,18261,work,1,False,1,757,1070,2051556,home,,16,WALK_TRANSIT,1.0586461337073116,1651.0,1500.0,fastest +18151113,55338,19758,school,1,True,1,699,794,2268889,school,,7,WALK_TRANSIT,1.2437700951912316,1608.0,1652.0,cheapest +18151117,55338,19758,school,1,False,1,794,699,2268889,home,,10,SHARED2FREE,0.9734243621086657,,, +18151505,55339,19758,work,1,True,1,762,794,2268938,work,,5,DRIVEALONEFREE,-0.09751857931395859,,, +18151509,55339,19758,work,1,False,1,794,762,2268938,home,,16,DRIVEALONEFREE,-0.00758777225194223,,, +18990529,57897,20552,work,1,True,1,1070,829,2373816,work,,7,WALK_TRANSIT,2.167693637863771,1500.0,1754.0,fastest +18990533,57897,20552,work,1,False,1,829,1070,2373816,home,,20,SHARED2FREE,0.8134213565684749,,, +18990545,57898,20552,atwork,1,True,2,948,948,2373818,othmaint,7.505984009752515,11,WALK,-0.43394566102590587,,, +18990546,57898,20552,atwork,2,True,2,948,948,2373818,atwork,,11,WALK,-0.43394566102590587,,, +18990549,57898,20552,atwork,1,False,1,948,948,2373818,work,,13,WALK,-0.43394566102590587,,, +18990857,57898,20552,work,1,True,1,948,829,2373857,work,,11,DRIVEALONEFREE,-0.4249793219573366,,, +18990861,57898,20552,work,1,False,2,739,948,2373857,escort,11.536722530937924,17,DRIVEALONEFREE,-0.43515529700511774,,, +18990862,57898,20552,work,2,False,2,829,739,2373857,home,,17,DRIVEALONEFREE,-0.0991454272062134,,, +18991185,57899,20552,work,1,True,1,687,829,2373898,work,,7,WALK,0.5220452718494769,,, +18991189,57899,20552,work,1,False,1,829,687,2373898,home,,17,WALK,0.6195549538462863,,, +18991841,57901,20552,work,1,True,1,708,829,2373980,work,,6,DRIVEALONEFREE,-0.2156500818117738,,, +18991845,57901,20552,work,1,False,1,829,708,2373980,home,,17,DRIVEALONEFREE,-0.8860210329051176,,, +18991849,57901,20552,work,1,True,2,687,829,2373981,othmaint,13.312507041737813,19,WALK,1.2038039184191651,,, +18991850,57901,20552,work,2,True,2,708,687,2373981,work,,19,SHARED2FREE,0.5575710262749435,,, +18991853,57901,20552,work,1,False,1,829,708,2373981,home,,19,DRIVEALONEFREE,0.2682235453386703,,, +20510417,62531,21869,school,1,True,1,938,900,2563802,school,,20,TAXI,0.9896882699261176,,, +20510421,62531,21869,school,1,False,1,900,938,2563802,home,,20,SHARED3FREE,0.9332356383119612,,, +20510569,62532,21869,escort,1,True,1,647,900,2563821,escort,,6,SHARED2FREE,0.335383128230131,,, +20510573,62532,21869,escort,1,False,1,900,647,2563821,home,,7,DRIVEALONEFREE,0.3214092822815929,,, +20510897,62533,21869,escort,1,True,1,695,900,2563862,escort,,5,SHARED3FREE,0.7146320921116115,,, +20510901,62533,21869,escort,1,False,4,996,695,2563862,shopping,13.9010789936427,6,SHARED3FREE,0.8524136187385734,,, +20510902,62533,21869,escort,2,False,4,565,996,2563862,eatout,14.39637292108163,6,SHARED2FREE,0.5414028775934869,,, +20510903,62533,21869,escort,3,False,4,1099,565,2563862,escort,14.873416692380914,6,SHARED3FREE,0.5359149885298558,,, +20510904,62533,21869,escort,4,False,4,900,1099,2563862,home,,6,SHARED2FREE,0.8757485870243559,,, +20510905,62533,21869,escort,1,True,1,518,900,2563863,escort,,11,SHARED2FREE,0.40373478431782406,,, +20510909,62533,21869,escort,1,False,1,900,518,2563863,home,,11,SHARED3FREE,0.40947212130825034,,, +20510913,62533,21869,escort,1,True,1,844,900,2563864,escort,,14,SHARED2FREE,0.2145990810721203,,, +20510917,62533,21869,escort,1,False,1,900,844,2563864,home,,14,SHARED2FREE,0.21754216515589814,,, +20511025,62533,21869,othdiscr,1,True,1,1070,900,2563878,othdiscr,,11,WALK_TRANSIT,3.1051303239859553,1500.0,73.0,fastest +20511029,62533,21869,othdiscr,1,False,1,900,1070,2563878,home,,12,WALK_TRANSIT,1.157154797772182,73.0,1500.0,fastest +20511401,62534,21869,school,1,True,1,793,900,2563925,school,,8,SHARED3FREE,0.6297085882553292,,, +20511405,62534,21869,school,1,False,1,900,793,2563925,home,,9,SHARED3FREE,0.5963697573192646,,, +22303745,67999,23619,escort,1,True,1,767,973,2787968,escort,,13,DRIVEALONEFREE,0.890462913926255,,, +22303749,67999,23619,escort,1,False,3,1023,767,2787968,eatout,15.21145768299369,13,DRIVEALONEFREE,0.9380038576909149,,, +22303750,67999,23619,escort,2,False,3,993,1023,2787968,escort,16.38782485475755,13,DRIVEALONEFREE,1.1641324346709314,,, +22303751,67999,23619,escort,3,False,3,973,993,2787968,home,,13,SHARED3FREE,1.1940117377263124,,, +22303961,67999,23619,social,1,True,1,988,973,2787995,social,,17,WALK,-0.5648561131494263,,, +22303965,67999,23619,social,1,False,1,973,988,2787995,home,,20,WALK,-0.532826088016878,,, +22304313,68000,23619,work,1,True,2,929,973,2788039,social,22.324255397220206,7,SHARED3FREE,1.2410412266329416,,, +22304314,68000,23619,work,2,True,2,1070,929,2788039,work,,8,WALK_TRANSIT,3.2620308996092504,1500.0,1717.0,fastest +22304317,68000,23619,work,1,False,1,973,1070,2788039,home,,21,WALK_TRANSIT,1.5243469382674963,1588.0,1500.0,fastest +25904705,78977,26897,school,1,True,1,984,1081,3238088,school,,7,WALK,-0.4961040893656598,,, +25904709,78977,26897,school,1,False,1,1081,984,3238088,home,,14,WALK,-0.4863570793073567,,, +25905145,78979,26897,atwork,1,True,1,1070,897,3238143,atwork,,9,DRIVEALONEFREE,-0.07905790168873156,,, +25905149,78979,26897,atwork,1,False,1,897,1070,3238143,work,,11,DRIVEALONEFREE,0.08969911648121655,,, +25905425,78979,26897,work,1,True,1,897,1081,3238178,work,,7,DRIVEALONEFREE,0.05437050774730416,,, +25905429,78979,26897,work,1,False,1,1081,897,3238178,home,,18,DRIVEALONEFREE,0.22372783432698584,,, +421021769,1283602,435012,work,1,True,1,1078,521,52627721,work,,8,WALK_TRANSIT,1.8021163156603706,1584.0,1608.0,fastest +421021773,1283602,435012,work,1,False,1,521,1078,52627721,home,,18,WALK_TRANSIT,0.6837263082556,1238.0,1584.0,shortest +421108705,1283868,435278,atwork,1,True,1,1070,1070,52638588,atwork,,12,WALK,2.4750620100011926,,, +421108709,1283868,435278,atwork,1,False,1,1070,1070,52638588,work,,12,WALK,2.4750620100011926,,, +421108753,1283868,435278,eatout,1,True,1,1070,537,52638594,eatout,,18,WALK_TRANSIT,3.4599905231494112,1500.0,1558.0,shortest +421108757,1283868,435278,eatout,1,False,2,1023,1070,52638594,shopping,16.211897411935006,21,WALK_TRANSIT,1.8946666992967236,1664.0,1500.0,cheapest +421108758,1283868,435278,eatout,2,False,2,537,1023,52638594,home,,21,SHARED2FREE,0.48140592645817604,,, +421109017,1283868,435278,work,1,True,1,1070,537,52638627,work,,9,WALK_TRANSIT,2.556266300862197,1500.0,1558.0,cheapest +421109021,1283868,435278,work,1,False,1,537,1070,52638627,home,,18,SHARED3FREE,0.9987470907812958,,, +421134601,1283946,435356,work,1,True,1,1070,523,52641825,work,,7,WALK_TRANSIT,2.564484302870523,1500.0,1604.0,shortest +421134605,1283946,435356,work,1,False,1,523,1070,52641825,home,,19,TNC_SINGLE,1.0882426323388465,,, +421348457,1284598,436008,work,1,True,1,553,562,52668557,work,,9,WALK,0.34775876785101223,,, +421348461,1284598,436008,work,1,False,1,562,553,52668557,home,,19,WALK,0.35217449627894437,,, +421878553,1286215,437625,atwork,1,True,1,1077,606,52734819,atwork,,10,WALK_TRANSIT,1.081956268867972,1500.0,1562.0,cheapest +421878557,1286215,437625,atwork,1,False,1,606,1077,52734819,work,,13,WALK_TRANSIT,1.6200392862281743,1562.0,1500.0,fastest +421878833,1286215,437625,work,1,True,1,606,656,52734854,work,,8,DRIVEALONEFREE,0.11024787576512153,,, +421878837,1286215,437625,work,1,False,3,934,606,52734854,othmaint,11.66765616556947,17,DRIVEALONEFREE,-0.33173180548126086,,, +421878838,1286215,437625,work,2,False,3,730,934,52734854,othmaint,11.283765093022993,19,DRIVEALONEFREE,-0.16433084991577782,,, +421878839,1286215,437625,work,3,False,3,656,730,52734854,home,,19,DRIVEALONEFREE,0.09340599142741139,,, +423180353,1290184,441594,atwork,1,True,1,732,977,52897544,atwork,,11,WALK,2.454959930326644,,, +423180357,1290184,441594,atwork,1,False,1,977,732,52897544,work,,11,SHARED2FREE,2.474151530078552,,, +423180401,1290184,441594,eatout,1,True,1,949,1096,52897550,eatout,,21,SHARED2FREE,0.029277265052942935,,, +423180405,1290184,441594,eatout,1,False,2,1070,949,52897550,othmaint,13.725541816697769,21,DRIVEALONEFREE,-0.5109700761493449,,, +423180406,1290184,441594,eatout,2,False,2,1096,1070,52897550,home,,21,WALK,1.478188695251212,,, +423180553,1290184,441594,othdiscr,1,True,1,684,1096,52897569,othdiscr,,5,DRIVEALONEFREE,-0.055255950841505565,,, +423180557,1290184,441594,othdiscr,1,False,1,1096,684,52897569,home,,7,TAXI,-0.01129391688099097,,, +423180561,1290184,441594,othdiscr,1,True,1,762,1096,52897570,othdiscr,,21,WALK_TRANSIT,1.6714757713025346,1748.0,1516.0,fastest +423180565,1290184,441594,othdiscr,1,False,1,1096,762,52897570,home,,22,SHARED2FREE,0.7128678537533746,,, +423180569,1290184,441594,othdiscr,1,True,1,725,1096,52897571,othdiscr,,23,WALK,-3.2668547335399403,,, +423180573,1290184,441594,othdiscr,1,False,1,1096,725,52897571,home,,23,WALK,-3.2660886555615467,,, +423180665,1290184,441594,work,1,True,1,977,1096,52897583,work,,9,DRIVEALONEFREE,-0.19686415987020087,,, +423180669,1290184,441594,work,1,False,2,996,977,52897583,shopping,10.57412967355008,20,WALK,0.20872113777343163,,, +423180670,1290184,441594,work,2,False,2,1096,996,52897583,home,,21,DRIVEALONEFREE,-0.1682790295025423,,, +423325641,1290626,442036,work,1,True,2,1070,1100,52915705,work,22.736072255128843,8,WALK_TRANSIT,2.680723362376414,1500.0,1584.0,cheapest +423325642,1290626,442036,work,2,True,2,1070,1070,52915705,work,,9,TAXI,2.9829663694028086,,, +423325645,1290626,442036,work,1,False,1,1100,1070,52915705,home,,18,WALK_TRANSIT,1.487658539961561,1584.0,1500.0,shortest +611033041,1862905,721960,othdiscr,1,True,1,986,960,76379130,othdiscr,,15,WALK_TRANSIT,1.2392000497595896,1618.0,1762.0,shortest +611033045,1862905,721960,othdiscr,1,False,1,960,986,76379130,home,,21,TAXI,1.030043298269935,,, +611033369,1862906,721960,othdiscr,1,True,1,992,960,76379171,othdiscr,,11,SHARED3FREE,0.8858713264739554,,, +611033373,1862906,721960,othdiscr,1,False,2,551,992,76379171,eatout,14.365988124910027,16,DRIVEALONEFREE,0.6551993708892542,,, +611033374,1862906,721960,othdiscr,2,False,2,960,551,76379171,home,,16,SHARED2FREE,0.5117735347645973,,, +647572569,1974306,760593,othdiscr,1,True,1,564,509,80946571,othdiscr,,8,WALK,0.12477528626857051,,, +647572573,1974306,760593,othdiscr,1,False,1,509,564,80946571,home,,16,WALK,0.11684501363456502,,, +647572729,1974307,760593,atwork,1,True,1,913,739,80946591,atwork,,12,SHARED3FREE,2.326963701825421,,, +647572733,1974307,760593,atwork,1,False,1,739,913,80946591,work,,13,SHARED3FREE,2.32846863119764,,, +647573009,1974307,760593,work,1,True,2,746,509,80946626,shopping,17.72989652110887,9,WALK_TRANSIT,1.3491672588128374,1748.0,1603.0,shortest +647573010,1974307,760593,work,2,True,2,739,746,80946626,work,,10,WALK_TRANSIT,1.9388432613606479,1662.0,1748.0,shortest +647573013,1974307,760593,work,1,False,2,560,739,80946626,work,18.21874805037896,17,WALK_TRANSIT,1.6522805849156224,1558.0,1748.0,shortest +647573014,1974307,760593,work,2,False,2,509,560,80946626,home,,18,SHARED3FREE,1.442812477343081,,, +647573097,1974308,760593,escort,1,True,1,716,509,80946637,escort,,5,SHARED3FREE,0.8908384736539703,,, +647573101,1974308,760593,escort,1,False,1,509,716,80946637,home,,5,SHARED3FREE,0.8921116412562625,,, +648387521,1976791,761445,escort,1,True,1,908,613,81048440,escort,,13,SHARED2FREE,0.43187460597514504,,, +648387525,1976791,761445,escort,1,False,1,613,908,81048440,home,,13,SHARED2FREE,0.42769386397548775,,, +648387809,1976792,761445,atwork,1,True,1,517,517,81048476,atwork,,9,WALK,3.011532825794776,,, +648387813,1976792,761445,atwork,1,False,1,517,517,81048476,work,,9,WALK,3.011532825794776,,, +648387825,1976792,761445,eatout,1,True,1,648,613,81048478,eatout,,13,DRIVEALONEFREE,0.4724646811902892,,, +648387829,1976792,761445,eatout,1,False,1,613,648,81048478,home,,19,TAXI,0.5571365851477499,,, +648388089,1976792,761445,work,1,True,1,517,613,81048511,work,,5,TAXI,1.5969608894805951,,, +648388093,1976792,761445,work,1,False,1,613,517,81048511,home,,12,SHARED2FREE,1.19217144041821,,, +649042753,1978788,762159,social,1,True,1,763,961,81130344,social,,8,SHARED2FREE,0.3356357191681573,,, +649042757,1978788,762159,social,1,False,1,961,763,81130344,home,,20,SHARED2FREE,0.3505516593935055,,, +649043193,1978790,762159,escort,1,True,1,992,961,81130399,escort,,8,SHARED2FREE,0.5305214939922914,,, +649043197,1978790,762159,escort,1,False,1,961,992,81130399,home,,8,SHARED2FREE,0.5188517472186277,,, +649043433,1978790,762159,work,1,True,1,672,961,81130429,work,,8,DRIVEALONEFREE,-0.3341036694423639,,, +649043437,1978790,762159,work,1,False,1,961,672,81130429,home,,17,DRIVEALONEFREE,-0.33189664309585226,,, +649043761,1978791,762159,work,1,True,1,1070,961,81130470,work,,7,WALK_TRANSIT,2.58470148067105,1500.0,1762.0,fastest +649043765,1978791,762159,work,1,False,1,961,1070,81130470,home,,17,TAXI,1.0619367021498107,,, +819359665,2498047,922602,school,1,True,1,865,730,102419958,school,,9,TAXI,1.1163475417962474,,, +819359669,2498047,922602,school,1,False,1,730,865,102419958,home,,16,SHARED2FREE,1.8016199836098332,,, +819360057,2498048,922602,work,1,True,1,735,730,102420007,work,,7,SHARED3FREE,1.5809126110190068,,, +819360061,2498048,922602,work,1,False,1,730,735,102420007,home,,16,WALK_TRANSIT,1.880736952455109,1612.0,1592.0,shortest +819360385,2498049,922602,work,1,True,1,763,730,102420048,work,,6,WALK,-0.7841986127882443,,, +819360389,2498049,922602,work,1,False,1,730,763,102420048,home,,16,WALK,-0.8403247829423018,,, +860079225,2622192,952720,school,1,True,1,995,1025,107509903,school,,7,TNC_SINGLE,1.6419964172015382,,, +860079229,2622192,952720,school,1,False,1,1025,995,107509903,home,,14,SHARED3FREE,1.5389530621520078,,, +860079377,2622193,952720,escort,1,True,1,773,1025,107509922,escort,,6,SHARED3FREE,0.9453815876415103,,, +860079381,2622193,952720,escort,1,False,1,1025,773,107509922,home,,6,SHARED2FREE,0.9690872179986346,,, +860079529,2622193,952720,othmaint,1,True,1,550,1025,107509941,othmaint,,8,TAXI,1.5000974833736833,,, +860079533,2622193,952720,othmaint,1,False,1,1025,550,107509941,home,,15,WALK_TRANSIT,1.504929950475204,1666.0,1238.0,shortest +860079897,2622194,952720,shopping,1,True,1,989,1025,107509987,shopping,,9,WALK,1.6280881165461683,,, +860079901,2622194,952720,shopping,1,False,1,1025,989,107509987,home,,11,SHARED3FREE,1.5894514263370938,,, +860080273,2622195,952720,work,1,True,1,1021,1025,107510034,work,,8,DRIVEALONEFREE,0.13076087259157862,,, +860080277,2622195,952720,work,1,False,1,1025,1021,107510034,home,,17,DRIVEALONEFREE,0.1511979688414838,,, +933123249,2844887,1028031,work,1,True,1,845,846,116640406,work,,11,WALK,1.4058976376655379,,, +933123253,2844887,1028031,work,1,False,1,846,845,116640406,home,,23,SHARED2FREE,1.3490304072755472,,, +962301409,2933845,1048898,school,1,True,1,666,574,120287676,school,,12,WALK,-1.1062441750768535,,, +962301413,2933845,1048898,school,1,False,1,574,666,120287676,home,,21,WALK,-1.227339677708164,,, +962301737,2933846,1048898,school,1,True,1,515,574,120287717,school,,8,SHARED2FREE,-2.095040038174043,,, +962301741,2933846,1048898,school,1,False,1,574,515,120287717,home,,16,SHARED2FREE,-2.102822435414125,,, +962302017,2933847,1048898,othdiscr,1,True,1,657,574,120287752,othdiscr,,7,WALK_TRANSIT,1.3416568401967581,1238.0,1559.0,cheapest +962302021,2933847,1048898,othdiscr,1,False,2,1070,657,120287752,eatout,16.215971752056717,12,WALK_TRANSIT,3.482863279857927,3.0,1238.0,shortest +962302022,2933847,1048898,othdiscr,2,False,2,574,1070,120287752,home,,12,WALK_TRANSIT,0.9522919082686397,1559.0,1500.0,fastest +962302457,2933848,1048898,work,1,True,1,502,574,120287807,work,,6,WALK_TRANSIT,1.0919479644277714,1333.0,1559.0,fastest +962302461,2933848,1048898,work,1,False,1,574,502,120287807,home,,18,WALK_TRANSIT,1.0565178398416113,1559.0,1333.0,shortest +1055052265,3216622,1148260,univ,1,True,1,1074,1076,131881533,univ,,14,TAXI,0.8326272536348395,,, +1055052269,3216622,1148260,univ,1,False,1,1076,1074,131881533,home,,15,WALK,0.7538832012716944,,, diff --git a/activitysim/examples/placeholder_sandag/test/test_sandag.py b/activitysim/examples/placeholder_sandag/test/test_sandag.py index 58db5d2d31..1c4a15e3c8 100644 --- a/activitysim/examples/placeholder_sandag/test/test_sandag.py +++ b/activitysim/examples/placeholder_sandag/test/test_sandag.py @@ -3,6 +3,7 @@ import os import shutil import subprocess +import sys import pandas as pd import pandas.testing as pdt @@ -44,41 +45,59 @@ def data(): build_data() -def run_test(zone, multiprocess=False): +def run_test(zone, multiprocess=False, sharrow=False): def test_path(dirname): return os.path.join(os.path.dirname(__file__), dirname) def regress(zone): # ## regress tours - regress_tours_df = pd.read_csv( - test_path(f"regress/final_{zone}_zone_tours.csv") - ) - tours_df = pd.read_csv(test_path(f"output/final_{zone}_zone_tours.csv")) + if sharrow and os.path.isfile( + test_path(f"regress/final_{zone}_zone_tours_sh.csv") + ): + regress_tours_df = pd.read_csv( + test_path(f"regress/final_{zone}_zone_tours_sh.csv") + ) + else: + regress_tours_df = pd.read_csv( + test_path(f"regress/final_{zone}_zone_tours.csv") + ) + tours_df = pd.read_csv(test_path(f"output_{zone}/final_{zone}_zone_tours.csv")) tours_df.to_csv( test_path(f"regress/final_{zone}_zone_tours_last_run.csv"), index=False ) - print(f"regress tours") - pdt.assert_frame_equal(tours_df, regress_tours_df, rtol=1e-03) + print("regress tours") + pdt.assert_frame_equal( + tours_df, regress_tours_df, rtol=1e-03, check_dtype=False + ) # ## regress trips - regress_trips_df = pd.read_csv( - test_path(f"regress/final_{zone}_zone_trips.csv") - ) - trips_df = pd.read_csv(test_path(f"output/final_{zone}_zone_trips.csv")) + if sharrow and os.path.isfile( + test_path(f"regress/final_{zone}_zone_trips_sh.csv") + ): + regress_trips_df = pd.read_csv( + test_path(f"regress/final_{zone}_zone_trips_sh.csv") + ) + else: + regress_trips_df = pd.read_csv( + test_path(f"regress/final_{zone}_zone_trips.csv") + ) + trips_df = pd.read_csv(test_path(f"output_{zone}/final_{zone}_zone_trips.csv")) trips_df.to_csv( test_path(f"regress/final_{zone}_zone_trips_last_run.csv"), index=False ) - print(f"regress trips") - pdt.assert_frame_equal(trips_df, regress_trips_df, rtol=1e-03) + print("regress trips") + pdt.assert_frame_equal( + trips_df, regress_trips_df, rtol=1e-03, check_dtype=False + ) # run test file_path = os.path.join(os.path.dirname(__file__), "simulation.py") if zone == "2": - base_configs = psrc_example_path(f"configs") + base_configs = psrc_example_path("configs") else: - base_configs = mtc_example_path(f"configs") + base_configs = mtc_example_path("configs") run_args = [ "-c", @@ -90,13 +109,46 @@ def regress(zone): "-d", example_path(f"data_{zone}"), "-o", - test_path("output"), + test_path(f"output_{zone}"), ] if multiprocess: run_args = run_args + ["-s", "settings_mp.yaml"] - subprocess.run(["coverage", "run", "-a", file_path] + run_args, check=True) + if sharrow: + run_args = ["-c", test_path(f"configs_{zone}_sharrow")] + run_args + + try: + subprocess.run(["coverage", "run", "-a", file_path] + run_args, check=True) + except FileNotFoundError: + subprocess.run([sys.executable, file_path] + run_args, check=True) + from tempfile import TemporaryFile + from time import sleep + + with TemporaryFile() as outputstream: + env = os.environ.copy() + pythonpath = env.pop("PYTHONPATH", None) + process = subprocess.Popen( + args=[sys.executable, file_path] + run_args, + shell=True, + stdout=outputstream, + stderr=subprocess.STDOUT, + cwd=os.getcwd(), + env=env, + ) + while process.poll() is None: + where = outputstream.tell() + lines = outputstream.read() + if not lines: + # Adjust the sleep interval to your needs + sleep(0.25) + # make sure pointing to the last place we read + outputstream.seek(where) + else: + # Windows adds an extra carriage return and then chokes on + # it when displaying (or, as it were, not displaying) the + # output. So we give Windows a little helping hand. + print(lines.decode().replace("\r\n", "\n"), end="") regress(zone) @@ -109,6 +161,13 @@ def test_1_zone_mp(data): run_test(zone="1", multiprocess=True) +def test_1_zone_sharrow(data): + # Run both single and MP in one test function + # guarantees that compile happens in single + run_test(zone="1", multiprocess=False, sharrow=True) + run_test(zone="1", multiprocess=True, sharrow=True) + + def test_2_zone(data): run_test(zone="2", multiprocess=False) @@ -117,6 +176,13 @@ def test_2_zone_mp(data): run_test(zone="2", multiprocess=True) +def test_2_zone_sharrow(data): + # Run both single and MP in one test function + # guarantees that compile happens in single + run_test(zone="2", multiprocess=False, sharrow=True) + run_test(zone="2", multiprocess=True, sharrow=True) + + def test_3_zone(data): run_test(zone="3", multiprocess=False) @@ -125,15 +191,28 @@ def test_3_zone_mp(data): run_test(zone="3", multiprocess=True) +def test_3_zone_sharrow(data): + # Run both single and MP in one test function + # guarantees that compile happens in single + run_test(zone="3", multiprocess=False, sharrow=True) + run_test(zone="3", multiprocess=True, sharrow=True) + + if __name__ == "__main__": # call each test explicitly so we get a pass/fail for each build_data() run_test(zone="1", multiprocess=False) run_test(zone="1", multiprocess=True) + run_test(zone="1", multiprocess=False, sharrow=True) + run_test(zone="1", multiprocess=True, sharrow=True) run_test(zone="2", multiprocess=False) run_test(zone="2", multiprocess=True) + run_test(zone="2", multiprocess=False, sharrow=True) + run_test(zone="2", multiprocess=True, sharrow=True) run_test(zone="3", multiprocess=False) run_test(zone="3", multiprocess=True) + run_test(zone="3", multiprocess=False, sharrow=True) + run_test(zone="3", multiprocess=True, sharrow=True) diff --git a/activitysim/examples/prototype_arc/configs/legacy-1.0.4/logging.yaml b/activitysim/examples/prototype_arc/configs/legacy-1.0.4/logging.yaml new file mode 100644 index 0000000000..b2a89a10ef --- /dev/null +++ b/activitysim/examples/prototype_arc/configs/legacy-1.0.4/logging.yaml @@ -0,0 +1,53 @@ +# Config for logging +# ------------------ +# See http://docs.python.org/2.7/library/logging.config.html#configuration-dictionary-schema + +logging: + version: 1 + disable_existing_loggers: true + + + # Configuring the default (root) logger is highly recommended + root: + level: NOTSET + handlers: [console, logfile] + + loggers: + + activitysim: + level: INFO + handlers: [console, logfile] + propagate: false + + orca: + level: WARN + handlers: [console, logfile] + propagate: false + + handlers: + + logfile: + class: logging.FileHandler + filename: activitysim.log + mode: w + formatter: fileFormatter + level: NOTSET + + console: + class: logging.StreamHandler + stream: ext://sys.stdout + formatter: simpleFormatter + level: NOTSET + + formatters: + + simpleFormatter: + class: logging.Formatter + # format: '%(levelname)s - %(name)s - %(message)s' + format: '%(levelname)s - %(message)s' + datefmt: '%d/%m/%Y %H:%M:%S' + + fileFormatter: + class: logging.Formatter + format: '%(asctime)s - %(levelname)s - %(name)s - %(message)s' + datefmt: '%d/%m/%Y %H:%M:%S' diff --git a/activitysim/examples/prototype_arc/configs/legacy-1.0.4/network_los.yaml b/activitysim/examples/prototype_arc/configs/legacy-1.0.4/network_los.yaml new file mode 100644 index 0000000000..6c851bbbee --- /dev/null +++ b/activitysim/examples/prototype_arc/configs/legacy-1.0.4/network_los.yaml @@ -0,0 +1,15 @@ +# read cached skims (using numpy memmap) from output directory (memmap is faster than omx ) +read_skim_cache: False +# write memmapped cached skims to output directory after reading from omx, for use in subsequent runs +write_skim_cache: False + +zone_system: 1 + +# glob 'skims*.omx' will match one or more files: skims.omx, skims1.omx, skims2.omx... +taz_skims: skims*.omx + +skim_time_periods: + time_window: 1440 + period_minutes: 30 + periods: [0, 6, 14, 24, 32, 48] + labels: ['EA', 'AM', 'MD', 'PM', 'EV'] diff --git a/activitysim/examples/prototype_arc/configs/legacy-1.0.4/settings.yaml b/activitysim/examples/prototype_arc/configs/legacy-1.0.4/settings.yaml new file mode 100644 index 0000000000..474914f18e --- /dev/null +++ b/activitysim/examples/prototype_arc/configs/legacy-1.0.4/settings.yaml @@ -0,0 +1,208 @@ +# input tables +input_table_list: + - tablename: households + filename: households.csv + # The index column is set before keep_columns, + # so don't put index in keep columns + index_col: household_id + rename_columns: + maz: home_zone_id + np: hhsize + nwrkrs_esr: num_workers + keep_columns: + #- household_id + - home_zone_id + - hhsize + - num_workers + - hincp + #- hhincAdj + #- adjinc + #- veh + - hht + #- bld + #- type + - tablename: persons + filename: persons.csv + # The index column is set before keep_columns, + # so don't put index in keep columns + index_col: person_id + rename_columns: + maz: home_zone_id + sporder: PNUM + keep_columns: + #- person_id + - household_id + - home_zone_id + - PNUM + - agep + #- employed + - pecasOcc + - sex + - esr + - wkw + - wkhp + #-mil + - schg + #-schl + #-indp02 + #-indp07 + #-occp02 + #-occp10 + #-n + - tablename: land_use + filename: land_use.csv + # The index column is set before keep_columns, + # so don't put index in keep columns + index_col: zone_id +# rename_columns: +# + keep_columns: + #- TAZ +# - construc +# - manufac +# - TCU +# - wholesl + - retail +# - FIRE + - service +# - private +# - govt + - emp + - pop + - hshld + - univ + - acres +# - otherEmp +# - district + - PARKTOT + - PARKLNG + - PROPFREE + - PARKRATE + - areatype +# - county + - CBDFlag + - N11 + - N21 + - N22 + - N23 + - N313233 + - N42 + - N4445 + - N4849 + - N51 + - N52 + - N53 + - N54 + - N55 + - N56 + - N61 + - N62 + - N71 + - N72 + - N81 + - N92 + - EnrollDS + - EnrollPD + - I_PCTLT10K + - I_PCT10TO20 + - I_PCT20TO40 + - I_PCTGT40 + - RetailEmp30 + - PARKING_ZONE + +# - shadow pricing global switches + +# turn shadow_pricing on and off for all models (e.g. school and work) +# shadow pricing is deprecated for less than full samples +# use_shadow_pricing: True + +# chunk_size: + +# assume enough RAM to not chunk +chunk_training_mode: disabled + +models: + - initialize_landuse + - initialize_households + - compute_accessibility + - school_location + - workplace_location + - auto_ownership_simulate + - free_parking + - cdap_simulate + - mandatory_tour_frequency + - mandatory_tour_scheduling + - joint_tour_frequency + - joint_tour_composition + - joint_tour_participation + - joint_tour_destination + - joint_tour_scheduling + - non_mandatory_tour_frequency + - non_mandatory_tour_destination + - non_mandatory_tour_scheduling + - tour_mode_choice_simulate + - atwork_subtour_frequency + - atwork_subtour_destination + - atwork_subtour_scheduling + - atwork_subtour_mode_choice + - stop_frequency + - trip_purpose + - trip_destination + - trip_purpose_and_destination + - trip_scheduling_choice + - trip_departure_choice + - trip_mode_choice + - parking_location + - write_data_dictionary + - track_skim_usage + - write_trip_matrices + - write_tables + +# resume_after: + +multiprocess: False +fail_fast: True + +multiprocess_steps: + - name: mp_initialize_landuse + begin: initialize_landuse + - name: mp_accessibility + begin: compute_accessibility + num_processes: 5 + slice: + tables: + - accessibility + # don't slice any tables not explicitly listed above in slice.tables + except: True + - name: mp_households + begin: school_location + num_processes: 5 + slice: + tables: + - households + - persons + - name: mp_summarize + begin: write_tables + + +output_tables: + h5_store: False + action: include + prefix: final_ + tables: + - checkpoints + - accessibility + - land_use + - households + - persons + - tours + - trips + - joint_tour_participants + +trace_od: +trace_hh_id: + +min_value_of_time: 1 +max_value_of_time: 50 +distributed_vot_mu: 0.684 +distributed_vot_sigma: 0.85 \ No newline at end of file diff --git a/activitysim/examples/prototype_arc/configs/legacy-1.0.4/trip_destination_annotate_trips_preprocessor.csv b/activitysim/examples/prototype_arc/configs/legacy-1.0.4/trip_destination_annotate_trips_preprocessor.csv new file mode 100644 index 0000000000..0c6dbc79f0 --- /dev/null +++ b/activitysim/examples/prototype_arc/configs/legacy-1.0.4/trip_destination_annotate_trips_preprocessor.csv @@ -0,0 +1,13 @@ +Description,Target,Expression +#,, +,tour_mode,"reindex(tours.tour_mode, df.tour_id)" +,tour_mode_is_transit,(tour_mode == 'WALK_ALLTRN')|(tour_mode == 'WALK_PRMTRN')|(tour_mode == 'PNR_ALLTRN')|(tour_mode == 'PNR_PRMTRN')|(tour_mode == 'KNR_ALLTRN')|(tour_mode == 'KNR_PRMTRN') +,tour_mode_is_shared_ride,(tour_mode == 'SHARED2FREE')|(tour_mode == 'SHARED2PAY')|(tour_mode == 'SHARED3FREE')|(tour_mode == 'SHARED3PAY') +,_tod,"np.where(df.outbound,reindex_i(tours.start, df.tour_id),reindex_i(tours.end, df.tour_id))" +,trip_period,network_los.skim_time_period_label(_tod) +,is_joint,"reindex(tours.tour_category, df.tour_id)=='joint'" +#,is_CBD,"reindex(land_use.areatype, df.dest_zone_id) == 1" +#,,not needed as school is not chosen as an intermediate trip destination +#,_grade_school,"(df.primary_purpose == 'school') & reindex(persons.is_gradeschool, df.person_id)" +#,size_segment,"df.primary_purpose.where(df.primary_purpose != 'school', np.where(_grade_school,'gradeschool', 'highschool'))" +,tour_leg_dest,"np.where(df.outbound,reindex(tours.destination, df.tour_id), reindex(tours.origin, df.tour_id))" \ No newline at end of file diff --git a/activitysim/examples/prototype_arc/configs/logging.yaml b/activitysim/examples/prototype_arc/configs/logging.yaml index a30a25c829..1db43e9982 100644 --- a/activitysim/examples/prototype_arc/configs/logging.yaml +++ b/activitysim/examples/prototype_arc/configs/logging.yaml @@ -24,6 +24,18 @@ logging: handlers: [console, logfile] propagate: false + filelock: + level: WARN + + sharrow: + level: INFO + + blib2to3: + level: WARN + + black: + level: WARN + handlers: logfile: @@ -36,7 +48,7 @@ logging: console: class: logging.StreamHandler stream: ext://sys.stdout - formatter: simpleFormatter + formatter: elapsedFormatter level: NOTSET formatters: @@ -44,7 +56,7 @@ logging: simpleFormatter: class: logging.Formatter # format: '%(levelname)s - %(name)s - %(message)s' - format: '%(levelname)s - %(message)s' + format: '%(name)s - %(levelname)s - %(message)s' datefmt: '%d/%m/%Y %H:%M:%S' fileFormatter: @@ -52,3 +64,7 @@ logging: format: '%(asctime)s - %(levelname)s - %(name)s - %(message)s' datefmt: '%d/%m/%Y %H:%M:%S' + elapsedFormatter: + (): activitysim.core.tracing.ElapsedTimeFormatter + format: '[{elapsedTime}] {levelname:s}: {message:s}' + style: '{' diff --git a/activitysim/examples/prototype_arc/configs/mandatory_tour_frequency.csv b/activitysim/examples/prototype_arc/configs/mandatory_tour_frequency.csv index 4e7a9fef4c..b3ce0e5695 100644 --- a/activitysim/examples/prototype_arc/configs/mandatory_tour_frequency.csv +++ b/activitysim/examples/prototype_arc/configs/mandatory_tour_frequency.csv @@ -25,10 +25,10 @@ Can walk to work - Retired interaction,(ptype == 5) & (distance_to_work < 3),0,0 Can walk to school - University student interaction,(ptype == 3) & (distance_to_school < 3),0,0,0,0.7114,0 Can walk to school - Driving-age child interaction,(ptype == 6) & (distance_to_school < 3),0,0,0,0.7114,0 Can walk to school - Pre-driving age child who is in school interaction,((ptype == 7)|(ptype == 8)) & (distance_to_school < 3),0,0,0,0.7114,0 -Can walk to work or school - Full-time worker interaction,(ptype == 1) & (distance_to_work < 3 | distance_to_school < 3),0,0,0,0,0.1391 -Can walk to work or school - Part-time worker interaction,(ptype == 2) & (distance_to_work < 3 | distance_to_school < 3),0,0,0,0,0.1391 -Can walk to work or school - University student interaction,(ptype == 3) & (distance_to_work < 3 | distance_to_school < 3),0,0,0,0,0.1391 -Can walk to work or school - Driving-age child interaction,(ptype == 6) & (distance_to_work < 3 | distance_to_school < 3),0,0,0,0,0.1391 +Can walk to work or school - Full-time worker interaction,(ptype == 1) & ((distance_to_work < 3) | (distance_to_school < 3)),0,0,0,0,0.1391 +Can walk to work or school - Part-time worker interaction,(ptype == 2) & ((distance_to_work < 3) | (distance_to_school < 3)),0,0,0,0,0.1391 +Can walk to work or school - University student interaction,(ptype == 3) & ((distance_to_work < 3) | (distance_to_school < 3)),0,0,0,0,0.1391 +Can walk to work or school - Driving-age child interaction,(ptype == 6) & ((distance_to_work < 3) | (distance_to_school < 3)),0,0,0,0,0.1391 Round trip auto time to work - Full-time worker interaction,(ptype == 1) * roundtrip_auto_time_to_work,0,-0.003518,0,0,-0.003124 Round trip auto time to work - Part-time worker interaction,(ptype == 2) * roundtrip_auto_time_to_work,0,-0.003518,0,0,-0.003124 Round trip auto time to work - University student interaction,(ptype == 3) * roundtrip_auto_time_to_work,0,-0.003518,0,0,-0.003124 diff --git a/activitysim/examples/prototype_arc/configs/network_los.yaml b/activitysim/examples/prototype_arc/configs/network_los.yaml index 6c851bbbee..fca9b46219 100644 --- a/activitysim/examples/prototype_arc/configs/network_los.yaml +++ b/activitysim/examples/prototype_arc/configs/network_los.yaml @@ -4,9 +4,31 @@ read_skim_cache: False write_skim_cache: False zone_system: 1 +name: prototype_arc # glob 'skims*.omx' will match one or more files: skims.omx, skims1.omx, skims2.omx... -taz_skims: skims*.omx +taz_skims: + omx: skims*.omx + zarr: skims.zarr + zarr-digital-encoding: + - {regex: .*_IVT , joint_dict: true} + - {regex: .*_DISTANCE , joint_dict: true} + - {regex: .*_TIME , joint_dict: true} + - {regex: .*_WALK , joint_dict: true} + - {regex: .*_FAR , joint_dict: true} + - {regex: .*_LRT , joint_dict: true} + - {regex: .*_TOLL , joint_dict: true} + - {regex: .*_XPEN , joint_dict: true} + - {regex: .*_HRT , joint_dict: true} + - {regex: .*_IWAIT , joint_dict: true} + - {regex: .*_LOCAL , joint_dict: true} + - {regex: .*_BRT , joint_dict: true} + - {regex: .*_XBUS , joint_dict: true} + - {regex: .*_COMRAIL , joint_dict: true} + - {regex: .*_BRDS , joint_dict: true} + - {regex: .*_XWAIT , joint_dict: true} + - {regex: .*_DIST , joint_dict: true} + - {regex: .*_AUTO , joint_dict: true} skim_time_periods: time_window: 1440 diff --git a/activitysim/examples/prototype_arc/configs/non_mandatory_tour_destination.csv b/activitysim/examples/prototype_arc/configs/non_mandatory_tour_destination.csv index 0f3c647c11..171ac0f3e8 100644 --- a/activitysim/examples/prototype_arc/configs/non_mandatory_tour_destination.csv +++ b/activitysim/examples/prototype_arc/configs/non_mandatory_tour_destination.csv @@ -1,6 +1,6 @@ Description,Expression,escort,escortkids,escortnokids,shopping,eatout,othmaint,social,othdiscr Mode choice logsum,mode_choice_logsum,1,1,1,coef_shopping_mode_logsum,coef_eatout_mode_logsum,coef_othmaint_mode_logsum,1,1 -Intrazonal,@(df['zone_id']==df['home_zone_id']),0,0,0,0,0,0,0,0 +#Intrazonal,@(df['zone_id']==df['home_zone_id']),0,0,0,0,0,0,0,0 CBD area type,@df['is_CBD'],0,0,0,coef_shopping_cbd,coef_eatout_cbd,coef_othmaint_cbd,coef_social_cbd,0 Urban high area type,@df['is_urban'],0,0,0,coef_shopping_urban,coef_eatout_urban,coef_othmaint_urban,coef_social_urban,coef_othdiscr_urban "# CTRAMP expression has two 'Distance' coefficients. For example, 0.2553 and 0.0100 for Escort, so used their sum here",,,,,,,,, diff --git a/activitysim/examples/prototype_arc/configs/non_mandatory_tour_frequency.csv b/activitysim/examples/prototype_arc/configs/non_mandatory_tour_frequency.csv index 36da94ff90..2056cec3dc 100644 --- a/activitysim/examples/prototype_arc/configs/non_mandatory_tour_frequency.csv +++ b/activitysim/examples/prototype_arc/configs/non_mandatory_tour_frequency.csv @@ -139,8 +139,8 @@ util_presence_of_university_student_and_maintenance_tour,Dummy for Presence of U util_presence_of_driving_school_kid_and_maintenance_tour,Dummy for Presence of Driving School Kid (other than modeled person) & Maintenance tour ,has_driving_kid * othmaint,coef_presence_of_driving_school_kid_and_maintenance_tour,coef_presence_of_driving_school_kid_and_maintenance_tour,coef_presence_of_driving_school_kid_and_maintenance_tour,coef_presence_of_driving_school_kid_and_maintenance_tour,coef_presence_of_driving_school_kid_and_maintenance_tour,coef_presence_of_driving_school_kid_and_maintenance_tour,coef_presence_of_driving_school_kid_and_maintenance_tour,coef_presence_of_driving_school_kid_and_maintenance_tour util_presence_of_pre_driving_school_kid_and_maintenance_tour,Dummy for Presence of Pre-Driving School Kid (other than modeled person) & Maintenance tour ,has_school_kid * othmaint,coef_presence_of_pre_driving_school_kid_and_maintenance_tour,coef_presence_of_pre_driving_school_kid_and_maintenance_tour,coef_presence_of_pre_driving_school_kid_and_maintenance_tour,coef_presence_of_pre_driving_school_kid_and_maintenance_tour,coef_presence_of_pre_driving_school_kid_and_maintenance_tour,coef_presence_of_pre_driving_school_kid_and_maintenance_tour,coef_presence_of_pre_driving_school_kid_and_maintenance_tour,coef_presence_of_pre_driving_school_kid_and_maintenance_tour util_presence_of_pre_school_kid_and_maintenance_tour,Dummy for Presence of Pre-School Kid (other than modeled person) & Maintenance tour ,has_preschool_kid * othmaint,coef_presence_of_pre_school_kid_and_maintenance_tour,coef_presence_of_pre_school_kid_and_maintenance_tour,coef_presence_of_pre_school_kid_and_maintenance_tour,coef_presence_of_pre_school_kid_and_maintenance_tour,coef_presence_of_pre_school_kid_and_maintenance_tour,coef_presence_of_pre_school_kid_and_maintenance_tour,coef_presence_of_pre_school_kid_and_maintenance_tour,coef_presence_of_pre_school_kid_and_maintenance_tour -util_at_home_pre_driving_school_kid_and_maintenance_tour,Dummy for At home Pre-Driving School Kid & Maintenance tour ,has_school_kid_at_home * othmaint,coef_at_home_pre_driving_school_kid_and_maintenance_tour,coef_at_home_pre_driving_school_kid_and_maintenance_tour,coef_at_home_pre_driving_school_kid_and_maintenance_tour,coef_at_home_pre_driving_school_kid_and_maintenance_tour,coef_at_home_pre_driving_school_kid_and_maintenance_tour,coef_at_home_pre_driving_school_kid_and_maintenance_tour,coef_at_home_pre_driving_school_kid_and_maintenance_tour,coef_at_home_pre_driving_school_kid_and_maintenance_tour -util_at_home_pre_school_kid_and_maintenance_tour,Dummy for At homef Pre-School Kid & Maintenance tour ,has_preschool_kid_at_home * othmaint,coef_at_home_pre_school_kid_and_maintenance_tour,coef_at_home_pre_school_kid_and_maintenance_tour,coef_at_home_pre_school_kid_and_maintenance_tour,coef_at_home_pre_school_kid_and_maintenance_tour,coef_at_home_pre_school_kid_and_maintenance_tour,coef_at_home_pre_school_kid_and_maintenance_tour,coef_at_home_pre_school_kid_and_maintenance_tour,coef_at_home_pre_school_kid_and_maintenance_tour +#util_at_home_pre_driving_school_kid_and_maintenance_tour,Dummy for At home Pre-Driving School Kid & Maintenance tour ,has_school_kid_at_home * othmaint,coef_at_home_pre_driving_school_kid_and_maintenance_tour,coef_at_home_pre_driving_school_kid_and_maintenance_tour,coef_at_home_pre_driving_school_kid_and_maintenance_tour,coef_at_home_pre_driving_school_kid_and_maintenance_tour,coef_at_home_pre_driving_school_kid_and_maintenance_tour,coef_at_home_pre_driving_school_kid_and_maintenance_tour,coef_at_home_pre_driving_school_kid_and_maintenance_tour,coef_at_home_pre_driving_school_kid_and_maintenance_tour +#util_at_home_pre_school_kid_and_maintenance_tour,Dummy for At homef Pre-School Kid & Maintenance tour ,has_preschool_kid_at_home * othmaint,coef_at_home_pre_school_kid_and_maintenance_tour,coef_at_home_pre_school_kid_and_maintenance_tour,coef_at_home_pre_school_kid_and_maintenance_tour,coef_at_home_pre_school_kid_and_maintenance_tour,coef_at_home_pre_school_kid_and_maintenance_tour,coef_at_home_pre_school_kid_and_maintenance_tour,coef_at_home_pre_school_kid_and_maintenance_tour,coef_at_home_pre_school_kid_and_maintenance_tour util_presence_of_full_time_worker_and_eating_out_tour,Dummy for Presence of Full time Worker (other than modeled person) & Eating Out tour ,has_full_time * eatout,coef_presence_of_full_time_worker_and_eating_out_tour,coef_presence_of_full_time_worker_and_eating_out_tour,coef_presence_of_full_time_worker_and_eating_out_tour,coef_presence_of_full_time_worker_and_eating_out_tour,coef_presence_of_full_time_worker_and_eating_out_tour,coef_presence_of_full_time_worker_and_eating_out_tour,coef_presence_of_full_time_worker_and_eating_out_tour,coef_presence_of_full_time_worker_and_eating_out_tour util_presence_of_part_time_worker_and_eating_out_tour,Dummy for Presence of Part time Worker (other than modeled person) & Eating Out tour ,has_part_time * eatout,coef_presence_of_part_time_worker_and_eating_out_tour,coef_presence_of_part_time_worker_and_eating_out_tour,coef_presence_of_part_time_worker_and_eating_out_tour,coef_presence_of_part_time_worker_and_eating_out_tour,coef_presence_of_part_time_worker_and_eating_out_tour,coef_presence_of_part_time_worker_and_eating_out_tour,coef_presence_of_part_time_worker_and_eating_out_tour,coef_presence_of_part_time_worker_and_eating_out_tour util_presence_of_non_worker_and_eating_out_tour,Dummy for Presence of Non-Worker (other than modeled person) & Eating Out tour ,has_non_worker * eatout,coef_presence_of_non_worker_and_eating_out_tour,coef_presence_of_non_worker_and_eating_out_tour,coef_presence_of_non_worker_and_eating_out_tour,coef_presence_of_non_worker_and_eating_out_tour,coef_presence_of_non_worker_and_eating_out_tour,coef_presence_of_non_worker_and_eating_out_tour,coef_presence_of_non_worker_and_eating_out_tour,coef_presence_of_non_worker_and_eating_out_tour diff --git a/activitysim/examples/prototype_arc/configs/settings.yaml b/activitysim/examples/prototype_arc/configs/settings.yaml index 5dea2f6337..aa1c297ac3 100644 --- a/activitysim/examples/prototype_arc/configs/settings.yaml +++ b/activitysim/examples/prototype_arc/configs/settings.yaml @@ -5,6 +5,8 @@ input_table_list: # The index column is set before keep_columns, # so don't put index in keep columns index_col: household_id + recode_columns: + home_zone_id: land_use.zone_id rename_columns: maz: home_zone_id np: hhsize @@ -39,6 +41,8 @@ input_table_list: # The index column is set before keep_columns, # so don't put index in keep columns index_col: zone_id + recode_columns: + zone_id: zero-based keep_columns: - retail - service @@ -130,8 +134,8 @@ models: - track_skim_usage - write_trip_matrices - write_tables - -# resume_after: + +# resume_after: multiprocess: False fail_fast: True @@ -165,15 +169,29 @@ output_tables: tables: - checkpoints - accessibility - - land_use - - households - - persons - - tours - - trips + - tablename: land_use + decode_columns: + zone_id: land_use.zone_id + - tablename: households + decode_columns: + home_zone_id: land_use.zone_id + - tablename: persons + decode_columns: + home_zone_id: land_use.zone_id + school_zone_id: nonnegative | land_use.zone_id + workplace_zone_id: nonnegative | land_use.zone_id + - tablename: tours + decode_columns: + origin: land_use.zone_id + destination: land_use.zone_id + - tablename: trips + decode_columns: + origin: land_use.zone_id + destination: land_use.zone_id - joint_tour_participants - + #trace_od: [100,10] -trace_hh_id: 88 +trace_hh_id: # trace_hh_id: 2 #trace_hh_id: 1643904 @@ -181,3 +199,5 @@ min_value_of_time: 1 max_value_of_time: 50 distributed_vot_mu: 0.684 distributed_vot_sigma: 0.85 + +checkpoints: true diff --git a/activitysim/examples/prototype_arc/configs/settings_mp.yaml b/activitysim/examples/prototype_arc/configs/settings_mp.yaml index 103c07bbb6..5701b40290 100644 --- a/activitysim/examples/prototype_arc/configs/settings_mp.yaml +++ b/activitysim/examples/prototype_arc/configs/settings_mp.yaml @@ -5,6 +5,8 @@ input_table_list: # The index column is set before keep_columns, # so don't put index in keep columns index_col: household_id + recode_columns: + home_zone_id: land_use.zone_id rename_columns: maz: home_zone_id np: hhsize @@ -39,6 +41,8 @@ input_table_list: # The index column is set before keep_columns, # so don't put index in keep columns index_col: zone_id + recode_columns: + zone_id: zero-based keep_columns: - retail - service @@ -170,11 +174,25 @@ output_tables: tables: - checkpoints - accessibility - - land_use - - households - - persons - - tours - - trips + - tablename: land_use + decode_columns: + zone_id: land_use.zone_id + - tablename: households + decode_columns: + home_zone_id: land_use.zone_id + - tablename: persons + decode_columns: + home_zone_id: land_use.zone_id + school_zone_id: nonnegative | land_use.zone_id + workplace_zone_id: nonnegative | land_use.zone_id + - tablename: tours + decode_columns: + origin: land_use.zone_id + destination: land_use.zone_id + - tablename: trips + decode_columns: + origin: land_use.zone_id + destination: land_use.zone_id - joint_tour_participants diff --git a/activitysim/examples/prototype_arc/configs/tour_mode_choice.csv b/activitysim/examples/prototype_arc/configs/tour_mode_choice.csv index 9aee67ee94..2091f5c09b 100644 --- a/activitysim/examples/prototype_arc/configs/tour_mode_choice.csv +++ b/activitysim/examples/prototype_arc/configs/tour_mode_choice.csv @@ -254,8 +254,8 @@ SCHOOLBUS - Grade school pre-driver,"@np.where(df.is_school, (coef_schBus_PreDri SCHOOLBUS - Grade school driver,"@np.where(df.is_school, coef_schBus_Driver * df.is_driving_age, 0)",,,,,,,,,,,,,,,1 SCHOOLBUS - Rural/exurban destination,"@np.where(df.is_school, coef_schBus_Rural * df.is_rural_or_exurban, 0)",,,,,,,,,,,,,,,1 SCHOOLBUS - o/d distance greater than 10 miles,"@np.where(df.is_school, coef_schBus_odDistgre10Mile * (odt_skims['SOV_FREE_DISTANCE']>10), 0)",,,,,,,,,,,,,,,1 -SCHOOLBUS - Depart from home before 8 am (downstream model),"@np.where(df.is_school, todOut<2 , 0)",,,,,,,,,,,,,,, -SCHOOLBUS - Return to home before 5 pm (downstream model),"@np.where(df.is_school, todIn<6, 0)",,,,,,,,,,,,,,, +#SCHOOLBUS - Depart from home before 8 am (downstream model),"@np.where(df.is_school, todOut<2 , 0)",,,,,,,,,,,,,,, +#SCHOOLBUS - Return to home before 5 pm (downstream model),"@np.where(df.is_school, todIn<6, 0)",,,,,,,,,,,,,,, SCHOOLBUS - In-vehicle time (ERROR-all probabilities are zero for 1204 rows),"@np.where(df.is_school, coef_ivt*(odt_skims['SOV_FREE_DISTANCE']+dot_skims['SOV_FREE_DISTANCE'])*60/schoolBusSpeed, 0)",,,,,,,,,,,,,,,1 SCHOOLBUS - Out-of-vehicle time,"@np.where(df.is_school, coef_ovt*schoolBusOvt, 0)",,,,,,,,,,,,,,,1 #,,,,,,,,,,,,,,,, diff --git a/activitysim/examples/prototype_arc/configs/tour_scheduling_joint.csv b/activitysim/examples/prototype_arc/configs/tour_scheduling_joint.csv index ed7609cc6c..e4b3c7dc76 100644 --- a/activitysim/examples/prototype_arc/configs/tour_scheduling_joint.csv +++ b/activitysim/examples/prototype_arc/configs/tour_scheduling_joint.csv @@ -111,7 +111,7 @@ SHOPPING - Departure Constant: 10:00 AM - 10:30 AM,@((df.tour_category == 'joint SHOPPING - Departure Constant: 10:30 AM - 11:00 AM,@((df.tour_category == 'joint') & (df.tour_type == 'shopping') & (df.start==16)),-0.309421311 SHOPPING - Departure Constant: After 11:00 AM,@((df.tour_category == 'joint') & (df.tour_type == 'shopping') & (df.start>16)),-0.541073357 SHOPPING - Departure Constant: Shift for every 30 minutes after 11:30 am - Linear,"@np.where(((df.tour_category == 'joint') & (df.tour_type == 'shopping') & (df.start>17)), (np.where((df.start<12), np.minimum(12-df.start,7),0) + np.where((df.start>17), np.minimum(df.start-17,24), 0)), 0)",-0.072013428 -SHOPPING - Departure Constant: Shift for every 30 minutes after 11:30 am - Squared,"@np.where(((df.tour_category == 'joint') & (df.tour_type == 'shopping') & (df.start>17)), ((np.where((df.start<12), np.minimum(12-df.start,7), 0) + np.where((df.start>17), np.minimum(df.start-17,24), 0)) ** 2), 0)",-0.000653398 +SHOPPING - Departure Constant: Shift for every 30 minutes after 11:30 am - Squared,"@np.where(((df.tour_category == 'joint') & (df.tour_type == 'shopping') & (df.start>17)), ((np.where((df.start<12), np.minimum(12-df.start,7), 0) + np.where((df.start>17), np.minimum(df.start-17,24), 0)).astype(np.float32) ** 2), 0)",-0.000653398 SHOPPING - Arrival Constant: Shift for every 30 minutes before 12:00 pm - Linear,"@np.where(((df.tour_category == 'joint') & (df.tour_type == 'shopping') & (df.end<19)), (np.where ((df.end<19), np.minimum(19-df.end,10), 0) + np.where((df.end>38), np.minimum(df.end-38,5), 0)), 0)",-0.18376635 SHOPPING - Arrival Constant: Before 12:30 PM,@((df.tour_category == 'joint') & (df.tour_type == 'shopping') & (df.end<20)),-0.716195343 SHOPPING - Arrival Constant: 12:30 PM - 03:00 PM,@((df.tour_category == 'joint') & (df.tour_type == 'shopping') & ( df.end>=20) & (df.end<=24)),-0.502714001 @@ -176,7 +176,7 @@ MAINTENANCE - Departure Constant: 10:00 AM - 10:30 AM,@((df.tour_category == 'jo MAINTENANCE - Departure Constant: 10:30 AM - 11:00 AM,@((df.tour_category == 'joint') & (df.tour_type == 'othmaint') & (df.start==16)),-0.427568963 MAINTENANCE - Departure Constant: After 11:00 AM,@((df.tour_category == 'joint') & (df.tour_type == 'othmaint') & (df.start>16)),-0.520863411 MAINTENANCE - Departure Constant: Shift for every 30 minutes after 11:30 am - Linear,"@np.where(((df.tour_category == 'joint') & (df.tour_type == 'othmaint') & (df.start>17)), np.where((df.start<10), np.minimum(10-df.start,7), 0) + np.where((df.start>17), np.minimum(df.start-17,24), 0), 0)",0.042879095 -MAINTENANCE - Departure Constant: Shift for every 30 minutes after 11:30 am - Squared,"@np.where(((df.tour_category == 'joint') & (df.tour_type == 'othmaint') & (df.start>17)), ((np.where((df.start<10), np.minimum(10-df.start,7), 0) + np.where((df.start>17), np.minimum(df.start-17,24), 0)) ** 2), 0)",-0.003157293 +MAINTENANCE - Departure Constant: Shift for every 30 minutes after 11:30 am - Squared,"@np.where(((df.tour_category == 'joint') & (df.tour_type == 'othmaint') & (df.start>17)), ((np.where((df.start<10), np.minimum(10-df.start,7), 0) + np.where((df.start>17), np.minimum(df.start-17,24), 0)).astype(np.float32) ** 2), 0)",-0.003157293 MAINTENANCE - Arrival Constant: Shift for every 30 minutes before 10:00 am - Linear,"@np.where(((df.tour_category == 'joint') & (df.tour_type == 'othmaint') & (df.end<15)), (np.where((df.end<15), np.minimum(15-df.end,9), 0) + np.where((df.end>28), np.minimum(df.end-28,16), 0)), 0)",-0.226803619 MAINTENANCE - Arrival Constant: Before 10:30 AM,@((df.tour_category == 'joint') & (df.tour_type == 'othmaint') & (df.end<16)),-0.223212258 MAINTENANCE - Arrival Constant: 10:30 AM - 11:00 AM,@((df.tour_category == 'joint') & (df.tour_type == 'othmaint') & (df.end==16)),0 diff --git a/activitysim/examples/prototype_arc/configs/tour_scheduling_nonmandatory.csv b/activitysim/examples/prototype_arc/configs/tour_scheduling_nonmandatory.csv index 9749a6303d..4e52d3509b 100644 --- a/activitysim/examples/prototype_arc/configs/tour_scheduling_nonmandatory.csv +++ b/activitysim/examples/prototype_arc/configs/tour_scheduling_nonmandatory.csv @@ -103,7 +103,7 @@ util_shopping_departure_constant_10_am_to_10_30_am,SHOPPING - Departure Constant util_shopping_departure_constant_10_30_am_to_11_00_am,SHOPPING - Departure Constant: 10:30 AM - 11:00 AM,@((df.tour_type == 'shopping') & (df.start==16)),coef_shopping_departure_constant_10_30_am_to_11_00_am util_shopping_departure_constant_after_11_am,SHOPPING - Departure Constant: After 11:00 AM,@((df.tour_type == 'shopping') & (df.start>16)),coef_shopping_departure_constant_after_11_am util_shopping_departure_constant_shift_for_every_30_minutes_after_11_30_am_linear,SHOPPING - Departure Constant: Shift for every 30 minutes after 11:30 am - Linear,"@np.where(((df.tour_type == 'shopping') & (df.start>17)), (np.where((df.start<12), np.minimum(12-df.start,7),0) + np.where((df.start>17), np.minimum(df.start-17,24), 0)), 0)",coef_shopping_departure_constant_shift_for_every_30_minutes_after_11_30_am_linear -util_shopping_departure_constant_shift_for_every_30_minutes_after_11_30_am_squared,SHOPPING - Departure Constant: Shift for every 30 minutes after 11:30 am - Squared,"@np.where(((df.tour_type == 'shopping') & (df.start>17)), ((np.where((df.start<12), np.minimum(12-df.start,7), 0) + np.where((df.start>17), np.minimum(df.start-17,24), 0)) ** 2), 0)",coef_shopping_departure_constant_shift_for_every_30_minutes_after_11_30_am_squared +util_shopping_departure_constant_shift_for_every_30_minutes_after_11_30_am_squared,SHOPPING - Departure Constant: Shift for every 30 minutes after 11:30 am - Squared,"@np.where(((df.tour_type == 'shopping') & (df.start>17)), ((np.where((df.start<12), np.minimum(12-df.start,7), 0) + np.where((df.start>17), np.minimum(df.start-17,24), 0)).astype(np.float32) ** 2), 0)",coef_shopping_departure_constant_shift_for_every_30_minutes_after_11_30_am_squared util_shopping_arrival_constant_shift_for_every_30_minutes_before_12_pm_linear,SHOPPING - Arrival Constant: Shift for every 30 minutes before 12:00 pm - Linear,"@np.where(((df.tour_type == 'shopping') & (df.end<19)), (np.where ((df.end<19), np.minimum(19-df.end,10), 0) + np.where((df.end>38), np.minimum(df.end-38,5), 0)), 0)",coef_shopping_arrival_constant_shift_for_every_30_minutes_before_12_pm_linear util_shopping_arrival_constant_before_12_30_pm,SHOPPING - Arrival Constant: Before 12:30 PM,@((df.tour_type == 'shopping') & (df.end<20)),coef_shopping_arrival_constant_before_12_30_pm util_shopping_arrival_constant_12_30_pm_to_3_pm,SHOPPING - Arrival Constant: 12:30 PM - 03:00 PM,@((df.tour_type == 'shopping') & ( df.end>=20) & (df.end<=24)),coef_shopping_arrival_constant_12_30_pm_to_3_pm @@ -159,7 +159,7 @@ util_maintenance_departure_constant_10_am_to_10_30_am,MAINTENANCE - Departure Co util_maintenance_departure_constant_10_30_am_to_11_am,MAINTENANCE - Departure Constant: 10:30 AM - 11:00 AM,@((df.tour_type == 'othmaint') & (df.start==16)),coef_maintenance_departure_constant_10_30_am_to_11_am util_maintenance_departure_constant_after_11_am,MAINTENANCE - Departure Constant: After 11:00 AM,@((df.tour_type == 'othmaint') & (df.start>16)),coef_maintenance_departure_constant_after_11_am util_maintenance_departure_constant_shift_for_every_30_minutes_after_11_30_am_linear,MAINTENANCE - Departure Constant: Shift for every 30 minutes after 11:30 am - Linear,"@np.where(((df.tour_type == 'othmaint') & (df.start>17)), np.where((df.start<10), np.minimum(10-df.start,7), 0) + np.where((df.start>17), np.minimum(df.start-17,24), 0), 0)",coef_maintenance_departure_constant_shift_for_every_30_minutes_after_11_30_am_linear -util_maintenance_departure_constant_shift_for_every_30_minutes_after_11_30_am_squared,MAINTENANCE - Departure Constant: Shift for every 30 minutes after 11:30 am - Squared,"@np.where(((df.tour_type == 'othmaint') & (df.start>17)), ((np.where((df.start<10), np.minimum(10-df.start,7), 0) + np.where((df.start>17), np.minimum(df.start-17,24), 0)) ** 2), 0)",coef_maintenance_departure_constant_shift_for_every_30_minutes_after_11_30_am_squared +util_maintenance_departure_constant_shift_for_every_30_minutes_after_11_30_am_squared,MAINTENANCE - Departure Constant: Shift for every 30 minutes after 11:30 am - Squared,"@np.where(((df.tour_type == 'othmaint') & (df.start>17)), ((np.where((df.start<10), np.minimum(10-df.start,7), 0) + np.where((df.start>17), np.minimum(df.start-17,24), 0)).astype(np.float32) ** 2), 0)",coef_maintenance_departure_constant_shift_for_every_30_minutes_after_11_30_am_squared util_maintenance_arrival_constant_shift_for_every_30_minutes_before_10_am_linear,MAINTENANCE - Arrival Constant: Shift for every 30 minutes before 10:00 am - Linear,"@np.where(((df.tour_type == 'othmaint') & (df.end<15)), (np.where((df.end<15), np.minimum(15-df.end,9), 0) + np.where((df.end>28), np.minimum(df.end-28,16), 0)), 0)",coef_maintenance_arrival_constant_shift_for_every_30_minutes_before_10_am_linear util_maintenance_arrival_constant_before_10_30_am,MAINTENANCE - Arrival Constant: Before 10:30 AM,@((df.tour_type == 'othmaint') & (df.end<16)),coef_maintenance_arrival_constant_before_10_30_am util_maintenance_arrival_constant_10_30_am_to_11_am,MAINTENANCE - Arrival Constant: 10:30 AM - 11:00 AM,@((df.tour_type == 'othmaint') & (df.end==16)),coef_maintenance_arrival_constant_10_30_am_to_11_am diff --git a/activitysim/examples/prototype_arc/configs/tour_scheduling_school.csv b/activitysim/examples/prototype_arc/configs/tour_scheduling_school.csv index b50c833380..63b3f3fb38 100644 --- a/activitysim/examples/prototype_arc/configs/tour_scheduling_school.csv +++ b/activitysim/examples/prototype_arc/configs/tour_scheduling_school.csv @@ -57,4 +57,4 @@ SCHOOL - Duration Constant: 8.5 hours,@(df.duration==17),-0.139412248 SCHOOL - Duration Constant: 9 hours,@(df.duration==18),-0.509620713 SCHOOL - Duration Constant: Longer than 9 hrs,@(df.duration>18),-0.561449384 SCHOOL - Duration Constant: Shift for every 30 minutes more than 9.5 hrs - Linear,"@(df.duration>19) * ((np.minimum(13-df.duration,47)*(df.duration<13)) + (np.minimum(df.duration-19,9)*(df.duration>19)))",0.379484906 -SCHOOL - Duration Constant: Shift for every 30 minutes more than 9.5 hrs - Squared,"@(df.duration>19) * (((np.minimum(13-df.duration,47)*(df.duration<13)) + (np.minimum(df.duration-19,9)*(df.duration>19))) ** 2)",-0.028814477 +SCHOOL - Duration Constant: Shift for every 30 minutes more than 9.5 hrs - Squared,"@(df.duration>19) * (((np.minimum(13-df.duration,47)*(df.duration<13)) + (np.minimum(df.duration-19,9)*(df.duration>19))).astype(np.float32) ** 2)",-0.028814477 diff --git a/activitysim/examples/prototype_arc/configs/trip_destination.csv b/activitysim/examples/prototype_arc/configs/trip_destination.csv index dc92ce84e1..5171258426 100644 --- a/activitysim/examples/prototype_arc/configs/trip_destination.csv +++ b/activitysim/examples/prototype_arc/configs/trip_destination.csv @@ -8,8 +8,8 @@ All - Tour origin zone,@(df.origin == df.dest_zone_id),0.7112,0.7112,0.7112,0.71 All - Tour destinatoin zone,@(df.destination == df.dest_zone_id),1.037,1.037,1.037,1.037,1.037,1.037,1.037,1.037,1.037,1.037 "# dest_zone_id is not available in the preprocessor file and reindex is not available here, so the expression was commented out. Similar expression was commented out in the ActivitySim example file as well. ",,,,,,,,,,, # All - CBD area type,"@reindex(land_use.areatype, df.dest_zone_id) == 1",-0.285,-0.285,-0.285,-0.285,-0.285,-0.285,-0.285,-0.285,-0.285,-0.285 -All - Size variable ,"@np.log1p(size_terms.get(df.dest_zone_id, df.purpose))",1,1,1,1,1,1,1,1,1,1 -All - No attractions,"@size_terms.get(df.dest_zone_id, df.purpose) == 0",-999,-999,-999,-999,-999,-999,-999,-999,-999,-999 +All - Size variable ,"@np.log1p(size_terms.get(df.dest_zone_id, df.purpose)) # sharrow: np.log1p(size_terms['arry'])",1,1,1,1,1,1,1,1,1,1 +All - No attractions,"@size_terms.get(df.dest_zone_id, df.purpose) == 0 # sharrow: size_terms['arry'] == 0",-999,-999,-999,-999,-999,-999,-999,-999,-999,-999 #Applied the conventions used in the destination sample file,,,,,,,,,,, ,"_org_to_stop_dist@od_skims[('SOV_FREE_DISTANCE', 'MD')]",1,1,1,1,1,1,1,1,1,1 ,"_stop_to_dest_dist@dp_skims[('SOV_FREE_DISTANCE', 'MD')]",1,1,1,1,1,1,1,1,1,1 diff --git a/activitysim/examples/prototype_arc/configs/trip_destination_annotate_trips_preprocessor.csv b/activitysim/examples/prototype_arc/configs/trip_destination_annotate_trips_preprocessor.csv index e94b9ea22d..c740b72fd3 100644 --- a/activitysim/examples/prototype_arc/configs/trip_destination_annotate_trips_preprocessor.csv +++ b/activitysim/examples/prototype_arc/configs/trip_destination_annotate_trips_preprocessor.csv @@ -10,4 +10,5 @@ Description,Target,Expression #,,not needed as school is not chosen as an intermediate trip destination #,_grade_school,"(df.primary_purpose == 'school') & reindex(persons.is_gradeschool, df.person_id)" #,size_segment,"df.primary_purpose.where(df.primary_purpose != 'school', np.where(_grade_school,'gradeschool', 'highschool'))" -,tour_leg_dest,"np.where(df.outbound,reindex(tours.destination, df.tour_id), reindex(tours.origin, df.tour_id))" \ No newline at end of file +,tour_leg_dest,"np.where(df.outbound,reindex(tours.destination, df.tour_id), reindex(tours.origin, df.tour_id))" +,purpose_index_num,"size_terms.get_cols(df.purpose)" \ No newline at end of file diff --git a/activitysim/examples/prototype_arc/configs/trip_destination_sample.csv b/activitysim/examples/prototype_arc/configs/trip_destination_sample.csv index 63b0031be3..b3a3627a1d 100644 --- a/activitysim/examples/prototype_arc/configs/trip_destination_sample.csv +++ b/activitysim/examples/prototype_arc/configs/trip_destination_sample.csv @@ -13,5 +13,5 @@ not available if walk tour not within walking distance,@(df.tour_mode=='WALK') & not available if bike tour not within biking distance,@(df.tour_mode=='BIKE') & (odt_skims['SOV_FREE_DISTANCE'] > max_bike_distance),-999,-999,-999,-999,-999,-999,-999,-999,-999,-999 not available if bike tour not within biking distance,@(df.tour_mode=='BIKE') & (dpt_skims['SOV_FREE_DISTANCE'] > max_bike_distance),-999,-999,-999,-999,-999,-999,-999,-999,-999,-999 "stop not available if tourMode is transit, and stop isn't walkable or accessible by transit","@np.where((((df.tour_mode=='WALK_ALLTRN')|(df.tour_mode=='WALK_PRMTRN')) & (((_od_AllIVT > 0) & (_od_DIST > 3) & (_dp_AllIVT > 0) & (_dp_DIST > 3)) | ((_od_AllIVT > 0) & (_od_DIST > 3) & (_dp_AllIVT > 0) & (_dp_DIST > 3)))), 1, 0)",-999,-999,-999,-999,-999,-999,-999,-999,-999,-999 -Size variable ,"@np.log1p(size_terms.get(df.dest_zone_id, df.purpose))",1,1,1,1,1,1,1,1,1,1 -No attractions,"@np.where(size_terms.get(df.dest_zone_id, df.purpose) == 0, 1, 0)",-999,-999,-999,-999,-999,-999,-999,-999,-999,-999 +Size variable ,"@np.log1p(size_terms.get(df.dest_zone_id, df.purpose)) # sharrow: np.log1p(size_terms['arry'])",1,1,1,1,1,1,1,1,1,1 +No attractions,"@np.where(size_terms.get(df.dest_zone_id, df.purpose) == 0, 1, 0) # sharrow: size_terms['arry'] == 0",-999,-999,-999,-999,-999,-999,-999,-999,-999,-999 diff --git a/activitysim/examples/prototype_arc/configs/trip_mode_choice_annotate_trips_preprocessor.csv b/activitysim/examples/prototype_arc/configs/trip_mode_choice_annotate_trips_preprocessor.csv index 3e55f253f2..8bab206a5d 100644 --- a/activitysim/examples/prototype_arc/configs/trip_mode_choice_annotate_trips_preprocessor.csv +++ b/activitysim/examples/prototype_arc/configs/trip_mode_choice_annotate_trips_preprocessor.csv @@ -13,8 +13,8 @@ free parking availability,free_parking_available,(df.tour_type == 'work') & df.f #cost coefficients,, ,c_cost,(c_cost_low * df.is_low_income)+(c_cost_med * df.is_medium_income)+(c_cost_high * df.is_high_income)+(c_cost_high * df.is_very_high_income) #,, -is_morning_period,_out_period,"(df.trip_period.isin(['EA','AM','MD']))" -is_afternoon_period,_in_period,"(df.trip_period.isin(['MD','PM']))" +is_morning_period,_out_period,"(df.trip_period.isin(['EA','AM','MD',0,1,2]))" +is_afternoon_period,_in_period,"(df.trip_period.isin(['MD','PM',2,3]))" #,, WLKALL transit in-vehicle time - outbound direction,walk_allTransit_ivt,odt_skims['WLK_ALLTRN_WLK_LOCAL']+expr*odt_skims['WLK_ALLTRN_WLK_XBUS']+hrt*odt_skims['WLK_ALLTRN_WLK_HRT']+brt*odt_skims['WLK_ALLTRN_WLK_BRT']+lrt*odt_skims['WLK_ALLTRN_WLK_LRT']+crl*odt_skims['WLK_ALLTRN_WLK_COMRAIL'] WLKPRM transit in-vehicle time - outbound direction,walk_prmTransit_ivt,odt_skims['WLK_PRMTRN_WLK_LOCAL']+expr*odt_skims['WLK_PRMTRN_WLK_XBUS']+hrt*odt_skims['WLK_PRMTRN_WLK_HRT']+brt*odt_skims['WLK_PRMTRN_WLK_BRT']+lrt*odt_skims['WLK_PRMTRN_WLK_LRT']+crl*odt_skims['WLK_PRMTRN_WLK_COMRAIL'] diff --git a/activitysim/examples/prototype_arc/configs/trip_scheduling_choice.csv b/activitysim/examples/prototype_arc/configs/trip_scheduling_choice.csv index 2933ca4956..7516a038e7 100644 --- a/activitysim/examples/prototype_arc/configs/trip_scheduling_choice.csv +++ b/activitysim/examples/prototype_arc/configs/trip_scheduling_choice.csv @@ -6,7 +6,7 @@ Discretionary tour-ASC for Legtime = 0,@(df['main_leg_duration'] == 0) & (df['to "Discretionary tour,ASC for Legtime = 3",@(df['main_leg_duration'] == 3) & (df['tour_type']=='othdiscr'),-1.0313 "Discretionary tour,ASC for Legtime = 4",@(df['main_leg_duration'] == 4) & (df['tour_type']=='othdiscr'),-0.46489 Discretionary tour - Main Leg time,"@np.where(df['tour_type']=='othdiscr', df['main_leg_duration'], 0)",0.060382 -Eatout tour - Shift,"@np.where(df['tour_type']=='eatout', np.where(df['main_leg_duration']> 4.5, (df['main_leg_duration'] - 4.5), (4.5 - df['main_leg_duration'])), 0)",-0.7508 +Eatout tour - Shift,"@np.where(df['tour_type']=='eatout', np.absolute(df['main_leg_duration'] - 4.5), 0)",-0.7508 Eatout tour - Main leg time,"@np.where(df['tour_type']=='eatout', df['main_leg_duration'], 0)",0.53247 "Maintenance tour,ASC for Legtime = 0",@(df['main_leg_duration'] == 0) & (df['tour_type']=='othmaint'),-3.6079 "Maintenance tour,ASC for Legtime = 1",@(df['main_leg_duration'] == 1) & (df['tour_type']=='othmaint'),-1.9376 @@ -78,8 +78,8 @@ Calibration,@(df['main_leg_duration'] == 0) & (df['tour_type']=='shopping'),-2.0 Calibration,@(df['main_leg_duration'] == 1) & (df['tour_type']=='shopping'),-1.0205 Calibration,@(df['main_leg_duration'] == 2) & (df['tour_type']=='shopping'),-0.0582 Calibration,@(df['main_leg_duration'] == 3) & (df['tour_type']=='shopping'),0.5533 -Calibration,@(df['main_leg_duration'] == 0) & (df['tour_type']=='eatout'),-100 -Calibration,@(df['main_leg_duration'] == 1) & (df['tour_type']=='eatout'),-100 +Calibration,@(df['main_leg_duration'] == 0) & (df['tour_type']=='eatout'),-50 +Calibration,@(df['main_leg_duration'] == 1) & (df['tour_type']=='eatout'),-50 Calibration,@(df['main_leg_duration'] == 2) & (df['tour_type']=='eatout'),-6.8372 Calibration,@(df['main_leg_duration'] == 3) & (df['tour_type']=='eatout'),-0.3319 Calibration,@(df['main_leg_duration'] == 4) & (df['tour_type']=='eatout'),0.8709 diff --git a/activitysim/examples/prototype_arc/test/configs/settings.yaml b/activitysim/examples/prototype_arc/test/configs/settings.yaml index d2b21a346f..52e1643451 100644 --- a/activitysim/examples/prototype_arc/test/configs/settings.yaml +++ b/activitysim/examples/prototype_arc/test/configs/settings.yaml @@ -38,3 +38,5 @@ output_tables: sort: True tables: - trips + +recode_pipeline_columns: False diff --git a/activitysim/examples/prototype_arc/test/configs_recode/network_los.yaml b/activitysim/examples/prototype_arc/test/configs_recode/network_los.yaml new file mode 100644 index 0000000000..1c4cd79daf --- /dev/null +++ b/activitysim/examples/prototype_arc/test/configs_recode/network_los.yaml @@ -0,0 +1,6 @@ +inherit_settings: True + +# read cached skims (using numpy memmap) from output directory (memmap is faster than omx ) +read_skim_cache: False +# write memmapped cached skims to output directory after reading from omx, for use in subsequent runs +write_skim_cache: False diff --git a/activitysim/examples/prototype_arc/test/configs_recode/settings.yaml b/activitysim/examples/prototype_arc/test/configs_recode/settings.yaml new file mode 100644 index 0000000000..7e9d6f92ac --- /dev/null +++ b/activitysim/examples/prototype_arc/test/configs_recode/settings.yaml @@ -0,0 +1,30 @@ +inherit_settings: True + +# EXAMPLE_HAS_WARNINGS +strict: False + +# number of households to simulate +households_sample_size: 10 +chunk_size: 0 + +# - shadow pricing global switches +use_shadow_pricing: False + +# turn writing of sample_tables on and off for all models +# (if True, tables will be written if DEST_CHOICE_SAMPLE_TABLE_NAME is specified in individual model settings) +want_dest_choice_sample_tables: False + +cleanup_pipeline_after_run: True + +output_tables: + h5_store: False + action: include + prefix: final_ + sort: True + tables: + - tablename: trips + decode_columns: + origin: land_use.zone_id + destination: land_use.zone_id + +recode_pipeline_columns: True diff --git a/activitysim/examples/prototype_arc/test/configs_sharrow/network_los.yaml b/activitysim/examples/prototype_arc/test/configs_sharrow/network_los.yaml new file mode 100644 index 0000000000..1c4cd79daf --- /dev/null +++ b/activitysim/examples/prototype_arc/test/configs_sharrow/network_los.yaml @@ -0,0 +1,6 @@ +inherit_settings: True + +# read cached skims (using numpy memmap) from output directory (memmap is faster than omx ) +read_skim_cache: False +# write memmapped cached skims to output directory after reading from omx, for use in subsequent runs +write_skim_cache: False diff --git a/activitysim/examples/prototype_arc/test/configs_sharrow/settings.yaml b/activitysim/examples/prototype_arc/test/configs_sharrow/settings.yaml new file mode 100644 index 0000000000..ad0541e5fc --- /dev/null +++ b/activitysim/examples/prototype_arc/test/configs_sharrow/settings.yaml @@ -0,0 +1,31 @@ +inherit_settings: True + +# EXAMPLE_HAS_WARNINGS +strict: False + +# number of households to simulate +households_sample_size: 10 +chunk_size: 0 + +# - shadow pricing global switches +use_shadow_pricing: False + +# turn writing of sample_tables on and off for all models +# (if True, tables will be written if DEST_CHOICE_SAMPLE_TABLE_NAME is specified in individual model settings) +want_dest_choice_sample_tables: False + +cleanup_pipeline_after_run: True + +output_tables: + h5_store: False + action: include + prefix: final_ + sort: True + tables: + - tablename: trips + decode_columns: + origin: land_use.zone_id + destination: land_use.zone_id + +sharrow: require +recode_pipeline_columns: True diff --git a/activitysim/examples/prototype_arc/test/regress/final_trips.csv b/activitysim/examples/prototype_arc/test/regress/final_trips.csv index 59d47d4b42..6cbfbff9d2 100644 --- a/activitysim/examples/prototype_arc/test/regress/final_trips.csv +++ b/activitysim/examples/prototype_arc/test/regress/final_trips.csv @@ -1,6 +1,6 @@ trip_id,person_id,household_id,primary_purpose,trip_num,outbound,trip_count,destination,origin,tour_id,purpose,destination_logsum,depart,trip_mode,mode_choice_logsum,parking_zone_id 37314161,113762,42730,othmaint,1,True,1,106,103,4664270,othmaint,,10,DRIVEALONEFREE,-0.3567815833091734,-1 -37314165,113762,42730,othmaint,1,False,1,103,106,4664270,home,,14,DRIVEALONEFREE,-0.3564603142738344,-1 +37314165,113762,42730,othmaint,1,False,1,103,106,4664270,home,,11,DRIVEALONEFREE,-0.3564603142738344,-1 38194977,116448,43843,atwork,1,True,1,106,101,4774372,atwork,,20,DRIVEALONEFREE,-0.3217517094135284,-1 38194981,116448,43843,atwork,1,False,1,101,106,4774372,work,,21,DRIVEALONEFREE,-0.3217517094135284,-1 38195065,116449,43843,othdiscr,1,True,1,106,103,4774383,othdiscr,,32,SHARED2FREE,0.7593914979829192,-1 @@ -23,58 +23,58 @@ trip_id,person_id,household_id,primary_purpose,trip_num,outbound,trip_count,dest 39614186,120774,45311,work,2,True,2,102,107,4951773,work,,11,DRIVEALONEFREE,-0.5816405543804167,-1 39614189,120774,45311,work,1,False,1,105,102,4951773,home,,30,DRIVEALONEFREE,-0.6046856504917144,-1 39614513,120775,45311,work,1,True,1,101,105,4951814,work,,9,DRIVEALONEFREE,-0.6009435992240908,-1 -39614517,120775,45311,work,1,False,3,101,101,4951814,work,11.024174872861991,26,DRIVEALONEFREE,-0.35670998854637154,-1 -39614518,120775,45311,work,2,False,3,111,101,4951814,othmaint,9.493269756810935,26,DRIVEALONEFREE,-0.6651734709739686,-1 +39614517,120775,45311,work,1,False,3,101,101,4951814,work,11.024174872861991,28,DRIVEALONEFREE,-0.35670998854637154,-1 +39614518,120775,45311,work,2,False,3,111,101,4951814,othmaint,9.493269756810935,28,DRIVEALONEFREE,-0.6651734709739686,-1 39614519,120775,45311,work,3,False,3,105,111,4951814,home,,29,DRIVEALONEFREE,-0.3629269920825958,-1 40387937,123133,46056,work,1,True,1,106,106,5048492,work,,20,DRIVEALONEFREE,-0.19777291302680963,-1 40387941,123133,46056,work,1,False,1,106,106,5048492,home,,40,DRIVEALONEFREE,-0.1974023956537246,-1 -43308361,132037,49258,othmaint,1,True,1,122,110,5413545,othmaint,,25,DRIVEALONEFREE,-0.7412541938781739,-1 -43308365,132037,49258,othmaint,1,False,2,114,122,5413545,eatout,8.583723975523268,27,DRIVEALONEFREE,-0.5182247403621675,-1 -43308366,132037,49258,othmaint,2,False,2,110,114,5413545,home,,29,DRIVEALONEFREE,-0.5957676920413972,-1 +43308361,132037,49258,othmaint,1,True,1,122,110,5413545,othmaint,,23,DRIVEALONEFREE,-0.7390051318168641,-1 +43308365,132037,49258,othmaint,1,False,2,114,122,5413545,eatout,8.6406219788818,24,DRIVEALONEFREE,-0.5175821724891663,-1 +43308366,132037,49258,othmaint,2,False,2,110,114,5413545,home,,24,DRIVEALONEFREE,-0.5938398692131043,-1 43308537,132038,49258,escort,1,True,1,107,110,5413567,escort,,10,SHARED3FREE,-0.002601425939803153,-1 43308541,132038,49258,escort,1,False,1,110,107,5413567,home,,23,SHARED3FREE,-0.002601425939803153,-1 44930737,136983,50912,work,1,True,2,123,112,5616342,eatout,9.550642761579045,31,DRIVEALONEFREE,-0.5493329919815063,-1 -44930738,136983,50912,work,2,True,2,104,123,5616342,work,,33,DRIVEALONEFREE,-0.5825075493812562,-1 -44930741,136983,50912,work,1,False,2,112,104,5616342,social,11.272661973962412,33,DRIVEALONEFREE,-0.5302670584678649,-1 +44930738,136983,50912,work,2,True,2,104,123,5616342,work,,32,DRIVEALONEFREE,-0.6666110144615174,-1 +44930741,136983,50912,work,1,False,2,112,104,5616342,social,11.272661973962412,34,DRIVEALONEFREE,-0.5302670584678649,-1 44930742,136983,50912,work,2,False,2,112,112,5616342,home,,34,DRIVEALONEFREE,-0.18331599397659298,-1 44931065,136984,50912,work,1,True,2,101,112,5616383,shopping,9.605913908355959,11,DRIVEALONEFREE,-0.6129478299617769,-1 44931066,136984,50912,work,2,True,2,107,101,5616383,work,,12,DRIVEALONEFREE,-0.6193944739341735,-1 44931069,136984,50912,work,1,False,3,124,107,5616383,work,10.993535935633657,28,DRIVEALONEFREE,-0.8496232702255249,-1 44931070,136984,50912,work,2,False,3,105,124,5616383,escort,9.834503858214005,29,DRIVEALONEFREE,-0.6786967541694641,-1 44931071,136984,50912,work,3,False,3,112,105,5616383,home,,30,DRIVEALONEFREE,-0.22867261297702796,-1 -47621473,145187,53716,othmaint,1,True,3,121,116,5952684,social,9.821407878228861,9,SHARED3FREE,-0.41955731333905055,-1 -47621474,145187,53716,othmaint,2,True,3,104,121,5952684,othmaint,9.184178106638575,9,SHARED3FREE,-0.7294908763885499,-1 -47621475,145187,53716,othmaint,3,True,3,122,104,5952684,othmaint,,9,SHARED3FREE,-0.7320178102493288,-1 -47621477,145187,53716,othmaint,1,False,1,116,122,5952684,home,,9,SHARED3FREE,-0.6156986440406667,-1 -47621737,145188,53716,escort,1,True,1,114,116,5952717,escort,,28,DRIVEALONEFREE,-0.15083797590032313,-1 -47621741,145188,53716,escort,1,False,1,116,114,5952717,home,,31,SHARED2FREE,-0.15179813514136692,-1 +47621473,145187,53716,othmaint,1,True,3,121,116,5952684,social,9.821407878228861,8,SHARED3FREE,-0.41955731333905055,-1 +47621474,145187,53716,othmaint,2,True,3,104,121,5952684,othmaint,9.184178106638575,11,SHARED3FREE,-0.7294908763885499,-1 +47621475,145187,53716,othmaint,3,True,3,122,104,5952684,othmaint,,11,SHARED3FREE,-0.7320178102493288,-1 +47621477,145187,53716,othmaint,1,False,1,116,122,5952684,home,,20,SHARED3FREE,-0.6134629528820534,-1 +47621737,145188,53716,escort,1,True,1,114,116,5952717,escort,,29,DRIVEALONEFREE,-0.15083797590032313,-1 +47621741,145188,53716,escort,1,False,1,116,114,5952717,home,,30,SHARED2FREE,-0.15179813514136692,-1 47622241,145189,53716,school,1,True,1,114,116,5952780,school,,10,SCHOOL_BUS,4.3079239998221395,-1 47622245,145189,53716,school,1,False,1,116,114,5952780,home,,24,SCHOOL_BUS,4.3079239998221395,-1 47622569,145190,53716,school,1,True,1,114,116,5952821,school,,9,SHARED2FREE,-0.20617904920050897,-1 47622573,145190,53716,school,1,False,1,116,114,5952821,home,,24,SHARED2FREE,-0.20568500752544042,-1 -48258513,147129,54342,othdiscr,1,True,1,116,117,6032314,othdiscr,,30,DRIVEALONEFREE,-0.5246167486667632,-1 -48258517,147129,54342,othdiscr,1,False,1,117,116,6032314,home,,32,DRIVEALONEFREE,-0.49859145340919503,-1 -48258537,147129,54342,othmaint,1,True,1,114,117,6032317,othmaint,,38,DRIVEALONEFREE,-0.6871321834564209,-1 -48258541,147129,54342,othmaint,1,False,2,114,114,6032317,shopping,9.168415036809055,40,DRIVEALONEFREE,-0.4237361037254333,-1 -48258542,147129,54342,othmaint,2,False,2,117,114,6032317,home,,40,DRIVEALONEFREE,-0.6845617927551271,-1 +48258513,147129,54342,othdiscr,1,True,1,116,117,6032314,othdiscr,,27,DRIVEALONEFREE,-0.5246167486667632,-1 +48258517,147129,54342,othdiscr,1,False,1,117,116,6032314,home,,33,DRIVEALONEFREE,-0.4912015503406525,-1 +48258537,147129,54342,othmaint,1,True,1,114,117,6032317,othmaint,,34,DRIVEALONEFREE,-0.6871321834564209,-1 +48258541,147129,54342,othmaint,1,False,2,114,114,6032317,shopping,9.168415036809055,37,DRIVEALONEFREE,-0.4237361037254333,-1 +48258542,147129,54342,othmaint,2,False,2,117,114,6032317,home,,38,DRIVEALONEFREE,-0.6845617927551271,-1 56357665,171822,63802,eatout,1,True,1,127,135,7044708,eatout,,31,DRIVEALONEFREE,-0.652624578666687,-1 56357669,171822,63802,eatout,1,False,1,135,127,7044708,home,,34,DRIVEALONEFREE,-0.6343104555130004,-1 56357689,171822,63802,escort,1,True,1,135,135,7044711,escort,,28,SHARED3FREE,0.07706324792840326,-1 56357693,171822,63802,escort,1,False,2,135,135,7044711,escort,11.335590369260917,28,SHARED3FREE,0.07706324792840326,-1 56357694,171822,63802,escort,2,False,2,135,135,7044711,home,,28,SHARED3FREE,0.07706324792840326,-1 56357737,171822,63802,othdiscr,1,True,3,131,135,7044717,othdiscr,12.08859943934242,13,SHARED2FREE,0.5999776535886836,-1 -56357738,171822,63802,othdiscr,2,True,3,130,131,7044717,shopping,13.322498757479453,13,SHARED2FREE,0.620004705610611,-1 -56357739,171822,63802,othdiscr,3,True,3,130,130,7044717,othdiscr,,13,SHARED2FREE,0.6960546331136191,-1 +56357738,171822,63802,othdiscr,2,True,3,130,131,7044717,shopping,13.322498757479453,14,SHARED2FREE,0.620004705610611,-1 +56357739,171822,63802,othdiscr,3,True,3,130,130,7044717,othdiscr,,14,SHARED2FREE,0.6960546331136191,-1 56357741,171822,63802,othdiscr,1,False,1,135,130,7044717,home,,14,SHARED2FREE,0.6487159219305315,-1 -56358209,171823,63802,shopping,1,True,4,131,135,7044776,othmaint,10.351547195188443,25,SHARED3FREE,-0.14873576359938484,-1 -56358210,171823,63802,shopping,2,True,4,131,131,7044776,social,12.414358478771883,26,SHARED3FREE,-0.01216970989402637,-1 -56358211,171823,63802,shopping,3,True,4,131,131,7044776,shopping,11.697701711651066,26,SHARED3FREE,-0.01216970989402637,-1 +56358209,171823,63802,shopping,1,True,4,131,135,7044776,othmaint,10.361561654862802,24,SHARED3FREE,-0.1461983375796215,-1 +56358210,171823,63802,shopping,2,True,4,131,131,7044776,social,12.296346896283627,25,SHARED3FREE,-0.01216970989402637,-1 +56358211,171823,63802,shopping,3,True,4,131,131,7044776,shopping,11.580712919708816,26,SHARED3FREE,-0.01216970989402637,-1 56358212,171823,63802,shopping,4,True,4,131,131,7044776,shopping,,26,SHARED3FREE,-0.01216970989402637,-1 -56358213,171823,63802,shopping,1,False,1,135,131,7044776,home,,26,DRIVEALONEFREE,-0.1509559426724583,-1 +56358213,171823,63802,shopping,1,False,1,135,131,7044776,home,,27,DRIVEALONEFREE,-0.1509559426724583,-1 56358473,171824,63802,othdiscr,1,True,1,131,135,7044809,othdiscr,,32,SHARED2FREE,-0.4602414968534174,-1 56358477,171824,63802,othdiscr,1,False,1,135,131,7044809,home,,37,SHARED2FREE,-0.45329299490631025,-1 56358521,171824,63802,school,1,True,2,135,135,7044815,escort,11.575429047908129,10,SHARED2FREE,0.10569338295856193,-1 -56358522,171824,63802,school,2,True,2,135,135,7044815,school,,17,SHARED3FREE,0.10569338295856193,-1 +56358522,171824,63802,school,2,True,2,135,135,7044815,school,,10,SHARED3FREE,0.10569338295856193,-1 56358525,171824,63802,school,1,False,2,135,135,7044815,othdiscr,11.816112681442535,25,SHARED3FREE,0.10545807870230886,-1 56358526,171824,63802,school,2,False,2,135,135,7044815,home,,26,SHARED3FREE,0.10545807870230886,-1 56358801,171825,63802,othdiscr,1,True,1,131,135,7044850,othdiscr,,29,SHARED3FREE,-0.2817694611797112,-1 diff --git a/activitysim/examples/prototype_arc/test/regress/final_trips_sh.csv b/activitysim/examples/prototype_arc/test/regress/final_trips_sh.csv new file mode 100644 index 0000000000..06466ae58c --- /dev/null +++ b/activitysim/examples/prototype_arc/test/regress/final_trips_sh.csv @@ -0,0 +1,91 @@ +trip_id,person_id,household_id,primary_purpose,trip_num,outbound,trip_count,destination,origin,tour_id,purpose,destination_logsum,depart,trip_mode,mode_choice_logsum,parking_zone_id +37314161,113762,42730,othmaint,1,True,1,106,103,4664270,othmaint,,10,DRIVEALONEFREE,-0.3567815721035004,-1 +37314165,113762,42730,othmaint,1,False,1,103,106,4664270,home,,11,DRIVEALONEFREE,-0.356460303068161,-1 +38194977,116448,43843,atwork,1,True,1,106,101,4774372,atwork,,20,DRIVEALONEFREE,-0.3217517137527465,-1 +38194981,116448,43843,atwork,1,False,1,101,106,4774372,work,,21,DRIVEALONEFREE,-0.3217517137527465,-1 +38195065,116449,43843,othdiscr,1,True,1,106,103,4774383,othdiscr,,32,SHARED2FREE,0.7593915111282219,-1 +38195069,116449,43843,othdiscr,1,False,1,103,106,4774383,home,,37,SHARED2FREE,0.7593915111282219,-1 +38195257,116448,43843,work,1,True,2,107,103,4774407,othmaint,9.4362755,10,DRIVEALONEFREE,-0.6671370863914491,-1 +38195258,116448,43843,work,2,True,2,101,107,4774407,work,,10,DRIVEALONEFREE,-0.5893840193748475,-1 +38195261,116448,43843,work,1,False,1,103,101,4774407,home,,30,DRIVEALONEFREE,-0.5012716650962832,-1 +38195585,116449,43843,work,1,True,2,107,103,4774448,othmaint,10.769853,12,DRIVEALONEFREE,-0.2947940896818454,-1 +38195586,116449,43843,work,2,True,2,102,107,4774448,work,,13,DRIVEALONEFREE,-0.2092975448438938,-1 +38195589,116449,43843,work,1,False,3,103,102,4774448,othmaint,10.96389,23,SHARED2FREE,0.09835210999240279,-1 +38195590,116449,43843,work,2,False,3,103,103,4774448,work,12.435295,24,DRIVEALONEFREE,0.24223826711784288,-1 +38195591,116449,43843,work,3,False,3,103,103,4774448,home,,26,DRIVEALONEFREE,0.2401515927071465,-1 +38195849,116450,43843,school,1,True,1,106,103,4774481,school,,9,SCHOOL_BUS,4.351044654846191,-1 +38195853,116450,43843,school,1,False,1,103,106,4774481,home,,27,SCHOOL_BUS,4.351044654846191,-1 +38195865,116450,43843,shopping,1,True,1,101,103,4774483,shopping,,27,SHARED2FREE,-0.4441019010696936,-1 +38195869,116450,43843,shopping,1,False,1,103,101,4774483,home,,30,SHARED2FREE,-0.45749089544283433,-1 +39613905,120774,45311,atwork,1,True,1,101,102,4951738,atwork,,20,DRIVEALONEFREE,-0.41128289699554466,-1 +39613909,120774,45311,atwork,1,False,1,102,101,4951738,work,,21,DRIVEALONEFREE,-0.41192552447319036,-1 +39614185,120774,45311,work,1,True,2,107,105,4951773,work,10.718042,10,DRIVEALONEFREE,-0.43097302317619324,-1 +39614186,120774,45311,work,2,True,2,102,107,4951773,work,,11,DRIVEALONEFREE,-0.5816405415534974,-1 +39614189,120774,45311,work,1,False,1,105,102,4951773,home,,30,DRIVEALONEFREE,-0.604685664176941,-1 +39614513,120775,45311,work,1,True,1,101,105,4951814,work,,9,DRIVEALONEFREE,-0.6009435653686525,-1 +39614517,120775,45311,work,1,False,3,101,101,4951814,work,11.024176,28,DRIVEALONEFREE,-0.3567099869251252,-1 +39614518,120775,45311,work,2,False,3,111,101,4951814,othmaint,9.49327,28,DRIVEALONEFREE,-0.6651734709739686,-1 +39614519,120775,45311,work,3,False,3,105,111,4951814,home,,29,DRIVEALONEFREE,-0.3629269599914551,-1 +40387937,123133,46056,work,1,True,1,106,106,5048492,work,,20,DRIVEALONEFREE,-0.19777289032936088,-1 +40387941,123133,46056,work,1,False,1,106,106,5048492,home,,40,DRIVEALONEFREE,-0.1974023878574371,-1 +43308361,132037,49258,othmaint,1,True,1,122,110,5413545,othmaint,,23,DRIVEALONEFREE,-0.7390050888061523,-1 +43308365,132037,49258,othmaint,1,False,2,114,122,5413545,eatout,8.640622,24,DRIVEALONEFREE,-0.517582178115845,-1 +43308366,132037,49258,othmaint,2,False,2,110,114,5413545,home,,24,DRIVEALONEFREE,-0.5938398838043213,-1 +43308537,132038,49258,escort,1,True,1,107,110,5413567,escort,,10,SHARED3FREE,-0.002601420005322437,-1 +43308541,132038,49258,escort,1,False,1,110,107,5413567,home,,24,SHARED3FREE,-0.002601420005322437,-1 +44930737,136983,50912,work,1,True,2,123,112,5616342,eatout,9.550643,31,DRIVEALONEFREE,-0.5493329763412476,-1 +44930738,136983,50912,work,2,True,2,104,123,5616342,work,,32,DRIVEALONEFREE,-0.6666110157966613,-1 +44930741,136983,50912,work,1,False,2,112,104,5616342,social,11.272662,34,DRIVEALONEFREE,-0.5302670001983643,-1 +44930742,136983,50912,work,2,False,2,112,112,5616342,home,,34,DRIVEALONEFREE,-0.18331599235534674,-1 +44931065,136984,50912,work,1,True,2,101,112,5616383,shopping,9.605914,11,DRIVEALONEFREE,-0.6129478216171266,-1 +44931066,136984,50912,work,2,True,2,107,101,5616383,work,,12,DRIVEALONEFREE,-0.6193944811820983,-1 +44931069,136984,50912,work,1,False,3,124,107,5616383,work,10.993536,28,DRIVEALONEFREE,-0.849623203277588,-1 +44931070,136984,50912,work,2,False,3,105,124,5616383,escort,9.834504,29,DRIVEALONEFREE,-0.6786966919898987,-1 +44931071,136984,50912,work,3,False,3,112,105,5616383,home,,30,DRIVEALONEFREE,-0.228672593832016,-1 +47621473,145187,53716,othmaint,1,True,3,121,116,5952684,social,9.821408,8,SHARED3FREE,-0.4195572993713509,-1 +47621474,145187,53716,othmaint,2,True,3,104,121,5952684,othmaint,9.184178,11,SHARED3FREE,-0.729490876197815,-1 +47621475,145187,53716,othmaint,3,True,3,122,104,5952684,othmaint,,11,SHARED3FREE,-0.7320178151130677,-1 +47621477,145187,53716,othmaint,1,False,1,116,122,5952684,home,,20,SHARED3FREE,-0.6134629858242939,-1 +47621737,145188,53716,escort,1,True,1,114,116,5952717,escort,,29,DRIVEALONEFREE,-0.1508379685964528,-1 +47621741,145188,53716,escort,1,False,1,116,114,5952717,home,,30,SHARED2FREE,-0.15179812895272474,-1 +47622241,145189,53716,school,1,True,1,114,116,5952780,school,,10,SCHOOL_BUS,4.307923793792725,-1 +47622245,145189,53716,school,1,False,1,116,114,5952780,home,,24,SCHOOL_BUS,4.307923793792725,-1 +47622569,145190,53716,school,1,True,1,114,116,5952821,school,,9,SHARED2FREE,-0.20617904275545368,-1 +47622573,145190,53716,school,1,False,1,116,114,5952821,home,,24,SHARED2FREE,-0.20568500108204935,-1 +48258513,147129,54342,othdiscr,1,True,1,116,117,6032314,othdiscr,,27,DRIVEALONEFREE,-0.5246167778968811,-1 +48258517,147129,54342,othdiscr,1,False,1,117,116,6032314,home,,33,DRIVEALONEFREE,-0.49120157957077043,-1 +48258537,147129,54342,othmaint,1,True,1,114,117,6032317,othmaint,,34,DRIVEALONEFREE,-0.6871321797370912,-1 +48258541,147129,54342,othmaint,1,False,2,114,114,6032317,shopping,9.168414,37,DRIVEALONEFREE,-0.42373609542846685,-1 +48258542,147129,54342,othmaint,2,False,2,117,114,6032317,home,,38,DRIVEALONEFREE,-0.6845617890357972,-1 +56357665,171822,63802,eatout,1,True,1,127,135,7044708,eatout,,31,DRIVEALONEFREE,-0.6526245474815369,-1 +56357669,171822,63802,eatout,1,False,1,135,127,7044708,home,,34,DRIVEALONEFREE,-0.6343104243278505,-1 +56357689,171822,63802,escort,1,True,1,135,135,7044711,escort,,28,SHARED3FREE,0.07706324286670248,-1 +56357693,171822,63802,escort,1,False,2,135,135,7044711,escort,11.335588,28,SHARED3FREE,0.07706324286670248,-1 +56357694,171822,63802,escort,2,False,2,135,135,7044711,home,,28,SHARED3FREE,0.07706324286670248,-1 +56357737,171822,63802,othdiscr,1,True,3,131,135,7044717,othdiscr,12.088599,13,SHARED2FREE,0.599977654783949,-1 +56357738,171822,63802,othdiscr,2,True,3,130,131,7044717,shopping,13.322498,14,SHARED2FREE,0.6200047250329783,-1 +56357739,171822,63802,othdiscr,3,True,3,130,130,7044717,othdiscr,,14,SHARED2FREE,0.6960546579187885,-1 +56357741,171822,63802,othdiscr,1,False,1,135,130,7044717,home,,14,SHARED2FREE,0.6487159186367744,-1 +56358209,171823,63802,shopping,1,True,4,131,135,7044776,othmaint,10.361562,24,SHARED3FREE,-0.14619837454037923,-1 +56358210,171823,63802,shopping,2,True,4,131,131,7044776,social,12.296347,25,SHARED3FREE,-0.012169709209450414,-1 +56358211,171823,63802,shopping,3,True,4,131,131,7044776,shopping,11.580712,26,SHARED3FREE,-0.012169709209450414,-1 +56358212,171823,63802,shopping,4,True,4,131,131,7044776,shopping,,26,SHARED3FREE,-0.012169709209450414,-1 +56358213,171823,63802,shopping,1,False,1,135,131,7044776,home,,27,DRIVEALONEFREE,-0.15095594351539895,-1 +56358473,171824,63802,othdiscr,1,True,1,131,135,7044809,othdiscr,,32,SHARED2FREE,-0.4602414938195248,-1 +56358477,171824,63802,othdiscr,1,False,1,135,131,7044809,home,,37,SHARED2FREE,-0.4532929919006896,-1 +56358521,171824,63802,school,1,True,2,135,135,7044815,escort,11.575429,10,SHARED2FREE,0.10569338088788002,-1 +56358522,171824,63802,school,2,True,2,135,135,7044815,school,,10,SHARED3FREE,0.10569338088788002,-1 +56358525,171824,63802,school,1,False,2,135,135,7044815,othdiscr,11.8161125,25,SHARED3FREE,0.10545807803885715,-1 +56358526,171824,63802,school,2,False,2,135,135,7044815,home,,26,SHARED3FREE,0.10545807803885715,-1 +56358801,171825,63802,othdiscr,1,True,1,131,135,7044850,othdiscr,,29,SHARED3FREE,-0.28176947749985704,-1 +56358805,171825,63802,othdiscr,1,False,2,132,131,7044850,social,10.476024,35,SHARED2FREE,-0.20277185632585107,-1 +56358806,171825,63802,othdiscr,2,False,2,135,132,7044850,home,,39,SHARED3FREE,-0.365217931623,-1 +56358809,171825,63802,othdiscr,1,True,4,135,135,7044851,othmaint,5.4133277,26,WALK,-0.7460585832595825,-1 +56358810,171825,63802,othdiscr,2,True,4,135,135,7044851,othmaint,5.44314,27,WALK,-0.7460585832595825,-1 +56358811,171825,63802,othdiscr,3,True,4,130,135,7044851,othmaint,5.3748727,28,WALK,-1.4660019874572756,-1 +56358812,171825,63802,othdiscr,4,True,4,130,130,7044851,othdiscr,,28,WALK,-0.7807589769363404,-1 +56358813,171825,63802,othdiscr,1,False,1,135,130,7044851,home,,28,WALK,-1.4660019874572756,-1 +56358849,171825,63802,school,1,True,1,135,135,7044856,school,,9,SHARED3FREE,0.10569338088788002,-1 +56358853,171825,63802,school,1,False,1,135,135,7044856,home,,24,SHARED3FREE,0.10569338088788002,-1 +56359177,171826,63802,school,1,True,1,135,135,7044897,school,,10,SHARED3FREE,0.10569338088788002,-1 +56359181,171826,63802,school,1,False,1,135,135,7044897,home,,22,SHARED3FREE,0.10569338088788002,-1 diff --git a/activitysim/examples/prototype_arc/test/test_arc.py b/activitysim/examples/prototype_arc/test/test_arc.py index 5f0c9baafd..5b35132496 100644 --- a/activitysim/examples/prototype_arc/test/test_arc.py +++ b/activitysim/examples/prototype_arc/test/test_arc.py @@ -2,6 +2,7 @@ # See full license in LICENSE.txt. import os import subprocess +import sys import pandas as pd import pandas.testing as pdt @@ -15,7 +16,7 @@ def teardown_function(func): inject.reinject_decorated_tables() -def test_arc(): +def _test_arc(recode=False, sharrow=False): def example_path(dirname): resource = os.path.join("examples", "prototype_arc", dirname) return pkg_resources.resource_filename("activitysim", resource) @@ -24,7 +25,11 @@ def test_path(dirname): return os.path.join(os.path.dirname(__file__), dirname) def regress(): - regress_trips_df = pd.read_csv(test_path("regress/final_trips.csv")) + if sharrow: + # sharrow results in tiny changes (one trip moving one time period earlier) + regress_trips_df = pd.read_csv(test_path("regress/final_trips_sh.csv")) + else: + regress_trips_df = pd.read_csv(test_path("regress/final_trips.csv")) final_trips_df = pd.read_csv(test_path("output/final_trips.csv")) # person_id,household_id,tour_id,primary_purpose,trip_num,outbound,trip_count,purpose, @@ -34,12 +39,30 @@ def regress(): file_path = os.path.join(os.path.dirname(__file__), "simulation.py") - subprocess.run( - [ - "coverage", - "run", - "-a", - file_path, + if recode: + run_args = [ + "-c", + test_path("configs_recode"), + "-c", + example_path("configs"), + "-d", + example_path("data"), + "-o", + test_path("output"), + ] + elif sharrow: + run_args = [ + "-c", + test_path("configs_sharrow"), + "-c", + example_path("configs"), + "-d", + example_path("data"), + "-o", + test_path("output"), + ] + else: + run_args = [ "-c", test_path("configs"), "-c", @@ -48,13 +71,30 @@ def regress(): example_path("data"), "-o", test_path("output"), - ], - check=True, - ) + ] + + if os.environ.get("GITHUB_ACTIONS") == "true": + subprocess.run(["coverage", "run", "-a", file_path] + run_args, check=True) + else: + subprocess.run([sys.executable, file_path] + run_args, check=True) regress() +def test_arc(): + _test_arc() + + +def test_arc_recode(): + _test_arc(recode=True) + + +def test_arc_sharrow(): + _test_arc(sharrow=True) + + if __name__ == "__main__": - test_arc() + _test_arc() + _test_arc(recode=True) + _test_arc(sharrow=True) diff --git a/activitysim/examples/prototype_mtc/configs/data_dictionary.yaml b/activitysim/examples/prototype_mtc/configs/data_dictionary.yaml new file mode 100644 index 0000000000..1fd9e670fe --- /dev/null +++ b/activitysim/examples/prototype_mtc/configs/data_dictionary.yaml @@ -0,0 +1,14 @@ + + +land_use: + county_id: + 1: San Francisco + 2: San Mateo + 3: Santa Clara + 4: Alameda + 5: Contra Costa + 6: Solano + 7: Napa + 8: Sonoma + 9: Marin + diff --git a/activitysim/examples/prototype_mtc/configs/legacy-1.0.4/logging.yaml b/activitysim/examples/prototype_mtc/configs/legacy-1.0.4/logging.yaml new file mode 100644 index 0000000000..3d62e8064b --- /dev/null +++ b/activitysim/examples/prototype_mtc/configs/legacy-1.0.4/logging.yaml @@ -0,0 +1,54 @@ +# Config for logging +# ------------------ +# See http://docs.python.org/2.7/library/logging.config.html#configuration-dictionary-schema + +logging: + version: 1 + disable_existing_loggers: true + + + # Configuring the default (root) logger is highly recommended + root: + level: NOTSET + handlers: [console, logfile] + + loggers: + + activitysim: + level: INFO + handlers: [console, logfile] + propagate: false + + orca: + level: WARNING + handlers: [console, logfile] + propagate: false + + handlers: + + logfile: + class: logging.FileHandler + filename: !!python/object/apply:activitysim.core.config.log_file_path ['activitysim.log'] + mode: w + formatter: fileFormatter + level: NOTSET + + console: + class: logging.StreamHandler + stream: ext://sys.stdout + formatter: simpleFormatter + level: !!python/object/apply:activitysim.core.mp_tasks.if_sub_task [WARNING, NOTSET] + + formatters: + + simpleFormatter: + class: logging.Formatter + format: !!python/object/apply:activitysim.core.mp_tasks.if_sub_task [ + '%(processName)-10s %(levelname)s - %(name)s - %(message)s', + '%(levelname)s - %(name)s - %(message)s'] + datefmt: '%d/%m/%Y %H:%M:%S' + + fileFormatter: + class: logging.Formatter + format: '%(asctime)s - %(levelname)s - %(name)s - %(message)s' + datefmt: '%d/%m/%Y %H:%M:%S' diff --git a/activitysim/examples/prototype_mtc/configs/legacy-1.0.4/network_los.yaml b/activitysim/examples/prototype_mtc/configs/legacy-1.0.4/network_los.yaml new file mode 100644 index 0000000000..ab3a7d5c67 --- /dev/null +++ b/activitysim/examples/prototype_mtc/configs/legacy-1.0.4/network_los.yaml @@ -0,0 +1,14 @@ +# read cached skims (using numpy memmap) from output directory (memmap is faster than omx ) +read_skim_cache: False +# write memmapped cached skims to output directory after reading from omx, for use in subsequent runs +write_skim_cache: False + +zone_system: 1 + +taz_skims: skims.omx + +skim_time_periods: + time_window: 1440 + period_minutes: 60 + periods: [0, 3, 5, 9, 14, 18, 24] # 3=3:00-3:59, 5=5:00-5:59, 9=9:00-9:59, 14=2:00-2:59, 18=6:00-6:59 + labels: ['EA', 'EA', 'AM', 'MD', 'PM', 'EV'] diff --git a/activitysim/examples/prototype_mtc/configs/legacy-1.0.4/settings.yaml b/activitysim/examples/prototype_mtc/configs/legacy-1.0.4/settings.yaml new file mode 100644 index 0000000000..40adb00f95 --- /dev/null +++ b/activitysim/examples/prototype_mtc/configs/legacy-1.0.4/settings.yaml @@ -0,0 +1,245 @@ +# input tables +# +# activitysim uses "well-known" index and foreign key names for imported tables (e.g. households, persons, land_use) +# as well as for created tables (tours, joint_tour_participants, trips) +# e.g. the households table must have an index column 'household_id' and the foreign key to households in the +# persons table is also household_id. This naming convention allows activitysim to intuit the relationship +# between tables - for instance, to ensure that multiprocess slicing includes all the persons, tours, and trips +# in the same subprocess pipeline. The same strategy is also when chunking choosers, and to support tracing by +# household_id. +# +# the input_table_list index_col directive instructs activitysim to set the imported table index to zone_id +# you cannot change the well-known name of the index by modifying this directive. However, if your input file +# has a different id column name, you can rename it to the required index name with the rename_columns directive. +# In the settings below, the 'TAZ' column in the imported table is renamed 'zone_id' in the rename_columns settings. +# +input_table_list: + # + # households (table index 'household_id') + # + - tablename: households + filename: households.csv + index_col: household_id + rename_columns: + HHID: household_id # household_id is the required index column + PERSONS: hhsize + workers: num_workers + VEHICL: auto_ownership + TAZ: home_zone_id + keep_columns: + - home_zone_id + - income + - hhsize + - HHT + - auto_ownership + - num_workers + # + # persons (table index 'person_id') + # + - tablename: persons + filename: persons.csv + index_col: person_id + rename_columns: + PERID: person_id # person_id is the required index column + keep_columns: + - household_id + - age + - PNUM + - sex + - pemploy + - pstudent + - ptype + # + # land_use (table index 'zone_id') + # + - tablename: land_use + filename: land_use.csv + index_col: zone_id + rename_columns: + TAZ: zone_id # person_id is the required index column + COUNTY: county_id + keep_columns: + - DISTRICT + - SD + - county_id + - TOTHH + - TOTPOP + - TOTACRE + - RESACRE + - CIACRE + - TOTEMP + - AGE0519 + - RETEMPN + - FPSEMPN + - HEREMPN + - OTHEMPN + - AGREMPN + - MWTEMPN + - PRKCST + - OPRKCST + - area_type + - HSENROLL + - COLLFTE + - COLLPTE + - TOPOLOGY + - TERMINAL + +# convert input CSVs to HDF5 format and save to outputs directory +# create_input_store: True + +#input_store: ../output/input_data.h5 + +# number of households to simulate +households_sample_size: 100 +# simulate all households +# households_sample_size: 0 + +# set false to disable variability check in simple_simulate and interaction_simulate +check_for_variability: False + +# - shadow pricing global switches + +# turn shadow_pricing on and off for all models (e.g. school and work) +# shadow pricing is deprecated for less than full samples +# see shadow_pricing.yaml for additional settings +use_shadow_pricing: False + +# turn writing of sample_tables on and off for all models +# (if True, tables will be written if DEST_CHOICE_SAMPLE_TABLE_NAME is specified in individual model settings) +want_dest_choice_sample_tables: False + +# log interaction simulate/sample expressions that return prohibitive utility values that exclude all alternatives +log_alt_losers: False + +# alternate dir to read/write cache (defaults to output_dir) +# used for skim cache, tvpb, and chunk_log caches +#cache_dir: data/cache + +############## +# +# chunking +# + +# chooser chunk size in gigabytes +# target top memory usage during activitysim run (including shared memory, loaded tables, and transient memory usage) +#chunk_size: 12_000_000_000 +chunk_size: 0 + +# minimum fraction of total chunk_size to reserve for adaptive chunking +min_available_chunk_ratio: 0.05 + +# initial number of chooser rows for first chunk in training mode +# when there is no pre-existing chunk_cache to set initial value +# ordinarily bigger is better as long as it is not so big it causes memory issues (e.g. accessibility with lots of zones) +default_initial_rows_per_chunk: 500 + +# method to calculate memory overhead when chunking is enabled +chunk_method: hybrid_uss + +# chunk training mode +# training to determine the chunking settings written to a cache file that is re-used for production runs +# training +# production +# disabled +# assume enough RAM to not chunk +chunk_training_mode: disabled + +# whether to preserve or delete subprocess chunk and mem logs when they are consolidated at end of multiprocess run +keep_chunk_logs: True +keep_mem_logs: True + +############## + +# - tracing + +# trace household id; comment out or leave empty for no trace +# households with all tour types +# [ 728370 1234067 1402924 1594625 1595333 1747572 1896849 1931818 2222690 2344951 2677154] +trace_hh_id: 982875 + +# trace origin, destination in accessibility calculation; comment out or leave empty for no trace +# trace_od: [5, 11] +trace_od: + + + +# to resume after last successful checkpoint, specify resume_after: _ +#resume_after: trip_destination +resume_after: + +checkpoints: True +# if checkpoints is False, no intermediate checkpoints will be written before the end of run +# (or if multiprocessing, at the end of each multiprocess_step) +#checkpoints: False +# explicit list of models to checkpoint +#checkpoints: +# - mandatory_tour_scheduling +# - non_mandatory_tour_scheduling +# - trip_mode_choice + + +models: + - initialize_landuse + - initialize_households + - compute_accessibility + - school_location + - workplace_location + - auto_ownership_simulate + - free_parking + - cdap_simulate + - mandatory_tour_frequency + - mandatory_tour_scheduling + - joint_tour_frequency + - joint_tour_composition + - joint_tour_participation + - joint_tour_destination + - joint_tour_scheduling + - non_mandatory_tour_frequency + - non_mandatory_tour_destination + - non_mandatory_tour_scheduling + - tour_mode_choice_simulate + - atwork_subtour_frequency + - atwork_subtour_destination + - atwork_subtour_scheduling + - atwork_subtour_mode_choice + - stop_frequency + - trip_purpose + - trip_destination + - trip_purpose_and_destination + - trip_scheduling + - trip_mode_choice + - write_data_dictionary + - track_skim_usage + - write_trip_matrices + - write_tables + +output_tables: + h5_store: False + action: include + prefix: final_ + tables: + - checkpoints + - accessibility + - land_use + - households + - persons + - tours + - trips + - joint_tour_participants + +# area_types less than this are considered urban +urban_threshold: 4 +cbd_threshold: 2 +rural_threshold: 6 + +# - value of time +min_value_of_time: 1 +max_value_of_time: 50 +distributed_vot_mu: 0.684 +distributed_vot_sigma: 0.85 + +household_median_value_of_time: + 1: 6.01 + 2: 8.81 + 3: 10.44 + 4: 12.86 \ No newline at end of file diff --git a/activitysim/examples/prototype_mtc/configs/legacy-1.0.4/trip_destination_annotate_trips_preprocessor.csv b/activitysim/examples/prototype_mtc/configs/legacy-1.0.4/trip_destination_annotate_trips_preprocessor.csv new file mode 100644 index 0000000000..9db6a951da --- /dev/null +++ b/activitysim/examples/prototype_mtc/configs/legacy-1.0.4/trip_destination_annotate_trips_preprocessor.csv @@ -0,0 +1,12 @@ +Description,Target,Expression +#,, +,tour_mode,"reindex(tours.tour_mode, df.tour_id)" +,_tod,"np.where(df.outbound,reindex_i(tours.start, df.tour_id),reindex_i(tours.end, df.tour_id))" +,trip_period,network_los.skim_time_period_label(_tod) +,is_joint,"reindex(tours.tour_category, df.tour_id)=='joint'" +#,,not needed as school is not chosen as an intermediate trip destination +#,_grade_school,"(df.primary_purpose == 'school') & reindex(persons.is_gradeschool, df.person_id)" +#,size_segment,"df.primary_purpose.where(df.primary_purpose != 'school', np.where(_grade_school,'gradeschool', 'highschool'))" +,tour_mode_is_walk,"reindex(tours.tour_mode, df.tour_id)=='WALK'" +,tour_mode_is_bike,"reindex(tours.tour_mode, df.tour_id)=='BIKE'" +,tour_leg_dest,"np.where(df.outbound,reindex(tours.destination, df.tour_id), reindex(tours.origin, df.tour_id)).astype(int)" diff --git a/activitysim/examples/prototype_mtc/configs/logging.yaml b/activitysim/examples/prototype_mtc/configs/logging.yaml index 43a70cb58e..3b2851ddd8 100644 --- a/activitysim/examples/prototype_mtc/configs/logging.yaml +++ b/activitysim/examples/prototype_mtc/configs/logging.yaml @@ -24,6 +24,18 @@ logging: handlers: [console, logfile] propagate: false + filelock: + level: WARN + + sharrow: + level: INFO + + blib2to3: + level: WARN + + black: + level: WARN + handlers: logfile: diff --git a/activitysim/examples/prototype_mtc/configs/mandatory_tour_frequency.csv b/activitysim/examples/prototype_mtc/configs/mandatory_tour_frequency.csv index 848bbf77aa..51094ee822 100644 --- a/activitysim/examples/prototype_mtc/configs/mandatory_tour_frequency.csv +++ b/activitysim/examples/prototype_mtc/configs/mandatory_tour_frequency.csv @@ -25,10 +25,10 @@ util_can_walk_to_work_retired,Can walk to work - Retired interaction,(ptype == 5 util_can_walk_to_school_univ,Can walk to school - University student interaction,(ptype == 3) & (distance_to_school < 3),,,,coef_can_walk_to_work_school2, util_can_walk_to_school_driving_age_child,Can walk to school - Driving-age child interaction,(ptype == 6) & (distance_to_school < 3),,,,coef_can_walk_to_work_school2, util_can_walk_to_school_pre_driving_age_child,Can walk to school - Pre-driving age child who is in school interaction,(ptype == 7) & (distance_to_school < 3),,,,coef_can_walk_to_work_school2, -util_can_walk_to_work_or_school_ft,Can walk to work or school - Full-time worker interaction,(ptype == 1) & (distance_to_work < 3 | distance_to_school < 3),,,,,coef_can_walk_to_work_and_school -util_can_walk_to_work_or_school_pt,Can walk to work or school - Part-time worker interaction,(ptype == 2) & (distance_to_work < 3 | distance_to_school < 3),,,,,coef_can_walk_to_work_and_school -util_can_walk_to_work_or_school_univ,Can walk to work or school - University student interaction,(ptype == 3) & (distance_to_work < 3 | distance_to_school < 3),,,,,coef_can_walk_to_work_and_school -util_can_walk_to_work_or_school_driving_age_child,Can walk to work or school - Driving-age child interaction,(ptype == 6) & (distance_to_work < 3 | distance_to_school < 3),,,,,coef_can_walk_to_work_and_school +util_can_walk_to_work_or_school_ft,Can walk to work or school - Full-time worker interaction,(ptype == 1) & ((distance_to_work < 3) | (distance_to_school < 3)),,,,,coef_can_walk_to_work_and_school +util_can_walk_to_work_or_school_pt,Can walk to work or school - Part-time worker interaction,(ptype == 2) & ((distance_to_work < 3) | (distance_to_school < 3)),,,,,coef_can_walk_to_work_and_school +util_can_walk_to_work_or_school_univ,Can walk to work or school - University student interaction,(ptype == 3) & ((distance_to_work < 3) | (distance_to_school < 3)),,,,,coef_can_walk_to_work_and_school +util_can_walk_to_work_or_school_driving_age_child,Can walk to work or school - Driving-age child interaction,(ptype == 6) & ((distance_to_work < 3) | (distance_to_school < 3)),,,,,coef_can_walk_to_work_and_school util_round_trip_auto_time_to_work_ft,Round trip auto time to work - Full-time worker interaction,(ptype == 1) * roundtrip_auto_time_to_work,,coef_round_trip_auto_time_to_work_work2,,,coef_round_trip_auto_time_to_work_school2 util_round_trip_auto_time_to_work_pt,Round trip auto time to work - Part-time worker interaction,(ptype == 2) * roundtrip_auto_time_to_work,,coef_round_trip_auto_time_to_work_work2,,,coef_round_trip_auto_time_to_work_school2 util_round_trip_auto_time_to_work_univ,Round trip auto time to work - University student interaction,(ptype == 3) * roundtrip_auto_time_to_work,,coef_round_trip_auto_time_to_work_work2,,,coef_round_trip_auto_time_to_work_school2 diff --git a/activitysim/examples/prototype_mtc/configs/network_los.yaml b/activitysim/examples/prototype_mtc/configs/network_los.yaml index eb5ce609fa..2e0309aa8a 100644 --- a/activitysim/examples/prototype_mtc/configs/network_los.yaml +++ b/activitysim/examples/prototype_mtc/configs/network_los.yaml @@ -4,8 +4,28 @@ read_skim_cache: False write_skim_cache: True zone_system: 1 +name: prototype_mtc -taz_skims: skims.omx +taz_skims: + omx: skims.omx + zarr: skims.zarr + zarr-digital-encoding: + - regex: ".*_BOARDS" + joint_dict: joined_board + - regex: ".*_FAR" + joint_dict: joined_fare + - regex: ".*_XWAIT" + joint_dict: joined_xwait + - regex: ".*_WAUX" + joint_dict: joined_waux + - regex: ".*_IWAIT" + joint_dict: joined_iwait + - regex: ".*_KEYIVT" + joint_dict: joined_keyivt + - regex: ".*_WAIT" + joint_dict: joined_wait + - regex: "DISTWALK|DISTBIKE" + joint_dict: joined_walkbike skim_time_periods: time_window: 1440 diff --git a/activitysim/examples/prototype_mtc/configs/settings.yaml b/activitysim/examples/prototype_mtc/configs/settings.yaml index 6b1258bcce..6577b94b95 100644 --- a/activitysim/examples/prototype_mtc/configs/settings.yaml +++ b/activitysim/examples/prototype_mtc/configs/settings.yaml @@ -26,6 +26,8 @@ input_table_list: workers: num_workers VEHICL: auto_ownership TAZ: home_zone_id + recode_columns: + home_zone_id: land_use.zone_id keep_columns: - home_zone_id - income @@ -58,6 +60,8 @@ input_table_list: rename_columns: TAZ: zone_id # person_id is the required index column COUNTY: county_id + recode_columns: + zone_id: zero-based keep_columns: - DISTRICT - SD @@ -90,7 +94,7 @@ input_table_list: #input_store: ../output/input_data.h5 # number of households to simulate -households_sample_size: 100 +households_sample_size: 100000 # simulate all households # households_sample_size: 0 @@ -155,7 +159,7 @@ keep_mem_logs: True # trace household id; comment out or leave empty for no trace # households with all tour types # [ 728370 1234067 1402924 1594625 1595333 1747572 1896849 1931818 2222690 2344951 2677154] -trace_hh_id: 982875 +trace_hh_id: # trace origin, destination in accessibility calculation; comment out or leave empty for no trace # trace_od: [5, 11] @@ -221,11 +225,25 @@ output_tables: tables: - checkpoints - accessibility - - land_use - - households - - persons - - tours - - trips + - tablename: land_use + decode_columns: + zone_id: land_use.zone_id + - tablename: households + decode_columns: + home_zone_id: land_use.zone_id + - tablename: persons + decode_columns: + home_zone_id: land_use.zone_id + school_zone_id: nonnegative | land_use.zone_id + workplace_zone_id: nonnegative | land_use.zone_id + - tablename: tours + decode_columns: + origin: land_use.zone_id + destination: land_use.zone_id + - tablename: trips + decode_columns: + origin: land_use.zone_id + destination: land_use.zone_id - joint_tour_participants # area_types less than this are considered urban @@ -244,3 +262,4 @@ household_median_value_of_time: 2: 8.81 3: 10.44 4: 12.86 + diff --git a/activitysim/examples/prototype_mtc/configs/tour_scheduling_joint.csv b/activitysim/examples/prototype_mtc/configs/tour_scheduling_joint.csv index 3c116bc9a6..f9946b9e60 100644 --- a/activitysim/examples/prototype_mtc/configs/tour_scheduling_joint.csv +++ b/activitysim/examples/prototype_mtc/configs/tour_scheduling_joint.csv @@ -29,8 +29,8 @@ util_adult_with_children_in_hh_arrive_19_21,Adult with children in HH - arrive 1 #,Mode Choice Logsum,mode_choice_logsum, #,, #,,, FIXME - use temps as timetable ops can be very time-consuming -util_dummy_adjacent_before,,"_adjacent_window_before@tt.adjacent_window_before(df.tour_id, df.start)",coef_dummy -util_dummy_adjacent_after,,"_adjacent_window_after@tt.adjacent_window_after(df.tour_id, df.end)",coef_dummy +util_dummy_adjacent_before,,"_adjacent_window_before@tt.adjacent_window_before(df.tour_id, df.start)>0",coef_dummy +util_dummy_adjacent_after,,"_adjacent_window_after@tt.adjacent_window_after(df.tour_id, df.end)>0",coef_dummy util_some_previously_scheduled_tour_ends_in_this_departure_hour,Some previously-scheduled tour ends in this departure hour,"@tt.previous_tour_ends(df.tour_id, df.start)",coef_some_previously_scheduled_tour_ends_in_this_departure_hour util_some_previously_scheduled_tour_begins_in_this_arrival_hour,Some previously-scheduled tour begins in this arrival hour,"@tt.previous_tour_begins(df.tour_id, df.end)",coef_some_previously_scheduled_tour_begins_in_this_arrival_hour util_adjacent_window_exists_before_this_departure_hour_first_tour_interaction,Adjacent window exists before this departure hour - first tour interaction,"@(df.tour_type_count>1) & (df.tour_type_num == 1) & _adjacent_window_before",coef_adjacent_window_exists_before_this_departure_hour_first_tour_interaction diff --git a/activitysim/examples/prototype_mtc/configs/tour_scheduling_nonmandatory.csv b/activitysim/examples/prototype_mtc/configs/tour_scheduling_nonmandatory.csv index e81bddda6e..46663b9502 100644 --- a/activitysim/examples/prototype_mtc/configs/tour_scheduling_nonmandatory.csv +++ b/activitysim/examples/prototype_mtc/configs/tour_scheduling_nonmandatory.csv @@ -1,12 +1,17 @@ Label,Description,Expression,Coefficient +is_escort,is_escort,_is_escort@df.tour_type == 'escort',0.000000000000001 +is_not_escort,is_not_escort,_is_not_escort@df.tour_type != 'escort',0.000000000000001 +is_shopping,is_shopping,_is_shopping@df.tour_type == 'shopping',0.000000000000001 +is_othmaint,is_othmaint,_is_othmaint@df.tour_type == 'othmaint',0.000000000000001 +is_social,is_social,_is_social@df.tour_type == 'social',0.000000000000001 util_subsequent_tour_must_start_after_previous_tour_for_this_purpose_ends,Subsequent tour must start after previous tour for this purpose ends,(start < end_previous) & (tour_type_num > 1),coef_subsequent_tour_must_start_after_previous_tour_for_this_purpose_ends util_free_flow_round_trip_auto_time_shift_effects_duration,Free-flow round trip auto time shift effects - duration,roundtrip_auto_time_to_work * duration,coef_free_flow_round_trip_auto_time_shift_effects_duration -util_shopping_tour_departure_shift_effects,Shopping tour - departure shift effects,(tour_type == 'shopping') * start,coef_shopping_tour_departure_shift_effects -util_shopping_tour_duration_shift_effects,Shopping tour - duration shift effects,(tour_type == 'shopping') * duration,coef_shopping_tour_duration_shift_effects -util_maintenance_tour_departure_shift_effects,Maintenance tour - departure shift effects,(tour_type == 'othmaint') * start,coef_maintenance_tour_departure_shift_effects -util_maintenance_tour_duration_shift_effects,Maintenance tour - departure shift effects,(tour_type == 'othmaint') * duration,coef_maintenance_tour_duration_shift_effects -util_visit_tour_departure_shift_effects_start,Visit tour - departure shift effects,(tour_type == 'social') * start,coef_visit_tour_departure_shift_effects -util_visit_tour_duration_shift_effects_duration,Visit tour - departure shift effects,(tour_type == 'social') * duration,coef_visit_tour_duration_shift_effects +util_shopping_tour_departure_shift_effects,Shopping tour - departure shift effects,@(_is_shopping) * df.start,coef_shopping_tour_departure_shift_effects +util_shopping_tour_duration_shift_effects,Shopping tour - duration shift effects,@(_is_shopping) * df.duration,coef_shopping_tour_duration_shift_effects +util_maintenance_tour_departure_shift_effects,Maintenance tour - departure shift effects,@(_is_othmaint) * df.start,coef_maintenance_tour_departure_shift_effects +util_maintenance_tour_duration_shift_effects,Maintenance tour - departure shift effects,@(_is_othmaint) * df.duration,coef_maintenance_tour_duration_shift_effects +util_visit_tour_departure_shift_effects_start,Visit tour - departure shift effects,@(_is_social) * df.start,coef_visit_tour_departure_shift_effects +util_visit_tour_duration_shift_effects_duration,Visit tour - departure shift effects,@(_is_social) * df.duration,coef_visit_tour_duration_shift_effects util_eat_out_tour_departure_shift_effects,Eat Out tour - departure shift effects,(tour_type == 'eatout') * start,coef_eat_out_tour_departure_shift_effects util_school_child_age_16_plus_departure_shift_effects,School child age 16+ - departure shift effects,(ptype == 6) * start,coef_school_child_age_16_plus_departure_shift_effects util_school_child_age_16_plus_duration_shift_effects,School child age 16+ - duration shift effects,(ptype == 6) * duration,coef_school_child_age_16_plus_duration_shift_effects @@ -19,19 +24,19 @@ util_number_of_escort_tours_departure_shift_effects,Number of escort tours - dep util_number_of_individual_non_mandatory_tours_excluding_escort_departure_shift_effects,Number of idividual non-mandatory tours (excluding escort) - departure shift effects,num_non_escort_tours * start,coef_number_of_individual_non_mandatory_tours_excluding_escort_departure_shift_effects util_first_of_2_plus_tours_for_same_purpose_departure_shift_effect,First of 2+ tours for same purpose - departure shift effect,((tour_type_count>1) & (tour_type_num == 1)) * start,coef_first_of_2_plus_tours_for_same_purpose_departure_shift_effect util_subsequent_of_2_plus_tours_for_same_purpose_duration_shift_effect,subsequent of 2+ tours for same purpose - duration shift effect,(tour_type_num > 1) * duration,coef_subsequent_of_2_plus_tours_for_same_purpose_duration_shift_effect -util_maintenance_tour_depart_before_7,Maintenance tour - depart before 7,(tour_type == 'othmaint') & (start < 7),coef_maintenance_tour_depart_before_7 -util_shopping_tour_depart_before_8,Shopping tour - depart before 8,(tour_type == 'shopping') & (start < 8),coef_shopping_tour_depart_before_8 -util_shopping_tour_arrive_after_22,Shopping tour - arrive after 22,(tour_type == 'shopping') & (end > 22),coef_shopping_tour_arrive_after_22 +util_maintenance_tour_depart_before_7,Maintenance tour - depart before 7,@(_is_othmaint) & (df.start < 7),coef_maintenance_tour_depart_before_7 +util_shopping_tour_depart_before_8,Shopping tour - depart before 8,@(_is_shopping) & (df.start < 8),coef_shopping_tour_depart_before_8 +util_shopping_tour_arrive_after_22,Shopping tour - arrive after 22,@(_is_shopping) & (df.end > 22),coef_shopping_tour_arrive_after_22 util_school_child_under_16_arrive_after_22,School child under 16 - arrive after 22,(ptype == 7) & (end > 22),coef_school_child_under_16_arrive_after_22 util_university_student_arrive_after_22,University student - arrive after 22,(ptype == 3) & (end > 22),coef_university_student_arrive_after_22 -util_shopping_tour_duration_lt_2_hours,Shopping tour - duration < 2 hours,(tour_type == 'shopping') & (duration < 2),coef_shopping_tour_duration_lt_2_hours +util_shopping_tour_duration_lt_2_hours,Shopping tour - duration < 2 hours,@(_is_shopping) & (df.duration < 2),coef_shopping_tour_duration_lt_2_hours util_discretionary_tour_duration_lt_2_hours,Discretionary tour - duration < 2 hours,(tour_type == 'othdiscr') & (duration < 2),coef_discretionary_tour_duration_lt_2_hours util_adult_with_children_in_hh_arrive_19_21,Adult with children in HH - arrive 19 - 21,adult & (num_children > 0) & ( end > 18 ) & ( end < 22 ),coef_adult_with_children_in_hh_arrive_19_21 #,,, #,Mode Choice Logsum,mode_choice_logsum,#mode_choice_logsum #,,,# -util_dummy_adjacent_before,,"_adjacent_window_before@tt.adjacent_window_before(df.person_id, df.start)",coef_dummy -util_dummy_adjacent_after,,"_adjacent_window_after@tt.adjacent_window_after(df.person_id, df.end)",coef_dummy +util_dummy_adjacent_before,,"_adjacent_window_before@tt.adjacent_window_before(df.person_id, df.start)>0",coef_dummy +util_dummy_adjacent_after,,"_adjacent_window_after@tt.adjacent_window_after(df.person_id, df.end)>0",coef_dummy util_some_previously_scheduled_tour_ends_in_this_departure_hour,Some previously-scheduled tour ends in this departure hour,"@tt.previous_tour_ends(df.person_id, df.start)",coef_some_previously_scheduled_tour_ends_in_this_departure_hour util_some_previously_scheduled_tour_begins_in_this_arrival_hour,Some previously-scheduled tour begins in this arrival hour,"@tt.previous_tour_begins(df.person_id, df.end)",coef_some_previously_scheduled_tour_begins_in_this_arrival_hour util_adjacent_window_exists_before_this_departure_hour_first_tour_interaction,Adjacent window exists before this departure hour - first tour interaction,"@(df.tour_type_count>1) & (df.tour_type_num == 1) & _adjacent_window_before",coef_adjacent_window_exists_before_this_departure_hour_first_tour_interaction @@ -40,57 +45,57 @@ util_adjacent_window_exists_before_this_departure_hour_second_plus_tour_interact util_adjacent_window_exists_after_this_arrival_hour_second_plus_tour_interaction,Adjacent window exists after this arrival hour - second+ tour interaction,"@(df.tour_type_num > 1) & _adjacent_window_after",coef_adjacent_window_exists_after_this_arrival_hour_second_plus_tour_interaction util_ratio_of_individual_non_mandatory_tours_to_be_scheduled_to_number_of_unscheduled_hours,Remaining individual non-mandatory tours to be scheduled / number of unscheduled hours,"@((1.0 + df.tour_count - df.tour_num)) / tt.remaining_periods_available(df.person_id, df.start, df.end)",coef_ratio_of_individual_non_mandatory_tours_to_be_scheduled_to_number_of_unscheduled_hours #,#,,# -util_departure_constants_early,Departure Constants -- Early (up to 5),(tour_type != 'escort') & (start < 6),coef_departure_constants_early -util_departure_constants_am_peak_1,Departure Constants -- AM peak 1 (6),(tour_type != 'escort') & (start == 6),coef_departure_constants_am_peak_1 -util_departure_constants_am_peak_2,Departure Constants -- AM peak 2 (7),(tour_type != 'escort') & (start == 7),coef_departure_constants_am_peak_2 -util_departure_constants_am_peak_3,Departure Constants -- AM peak 3 (8),(tour_type != 'escort') & (start == 8),coef_departure_constants_am_peak_3 -util_departure_constants_am_peak_4,Departure Constants -- AM peak 4 (9),(tour_type != 'escort') & (start == 9),coef_departure_constants_am_peak_4 -util_departure_constants_midday_1,Departure Constants -- Midday 1 (10 to 12),(tour_type != 'escort') & (start > 9) & (start < 13),coef_departure_constants_midday_1 -util_departure_constants_midday_2,Departure Constants -- Midday 2 (13 to 15),(tour_type != 'escort') & (start > 12) & (start < 16),coef_departure_constants_midday_2 -util_departure_constants_pm_peak,Departure Constants -- PM peak (16 to 18),(tour_type != 'escort') & (start > 15) & (start < 19),coef_departure_constants_pm_peak -util_departure_constants_evening,Departure Constants -- Evening (19 to 21),(tour_type != 'escort') & (start > 18) & (start < 22),coef_departure_constants_evening -util_departure_constants_late,Departure Constants -- Late (22 and later),(tour_type != 'escort') & (start > 21),coef_departure_constants_late -util_arrival_constants_early,Arrival Constants -- Early (up to 6),(tour_type != 'escort') & (end < 7),coef_arrival_constants_early -util_arrival_constants_am_peak,Arrival Constants -- AM peak (7 to 9),(tour_type != 'escort') & (end > 6) & (end < 10),coef_arrival_constants_am_peak -util_arrival_constants_midday_1,Arrival Constants -- Midday 1 (10 to 12),(tour_type != 'escort') & (end > 9) & (end < 13),coef_arrival_constants_midday_1 -util_arrival_constants_midday_2,Arrival Constants -- Midday 2 (13 to 14),(tour_type != 'escort') & (end > 12) & (end < 15),coef_arrival_constants_midday_2 -util_arrival_constants_pm_peak_1,Arrival Constants -- PM peak 1 (15),(tour_type != 'escort') & (end == 15),coef_arrival_constants_pm_peak_1 -util_arrival_constants_pm_peak_2,Arrival Constants -- PM peak 2 (16),(tour_type != 'escort') & (end == 16),coef_arrival_constants_pm_peak_2 -util_arrival_constants_pm_peak_3,Arrival Constants -- PM peak 3 (17),(tour_type != 'escort') & (end == 17),coef_arrival_constants_pm_peak_3 -util_arrival_constants_pm_peak_4,Arrival Constants -- PM peak 4 (18),(tour_type != 'escort') & (end == 18),coef_arrival_constants_pm_peak_4 -util_arrival_constants_evening,Arrival Constants -- Evening (19 to 21),(tour_type != 'escort') & (end > 18) & (end < 22),coef_arrival_constants_evening -util_arrival_constants_late,Arrival Constants -- Late (22 and later),(tour_type != 'escort') & (end > 21),coef_arrival_constants_late -util_duration_constants_0_to_1_hours,Duration Constants -- 0 to 1 hours,(tour_type != 'escort') & (duration < 2),coef_duration_constants_0_to_1_hours -util_duration_constants_2_to_3_hours,Duration Constants -- 2 to 3 hours,(tour_type != 'escort') & (duration > 1) & (duration < 4),coef_duration_constants_2_to_3_hours -util_duration_constants_4_to_5_hours,Duration Constants -- 4 to 5 hours,(tour_type != 'escort') & (duration > 3) & (duration < 6),coef_duration_constants_4_to_5_hours -util_duration_constants_6_to_7_hours,Duration Constants -- 6 to 7 hours,(tour_type != 'escort') & (duration > 5) & (duration < 8),coef_duration_constants_6_to_7_hours -util_duration_constants_8_to_10_hours,Duration Constants -- 8 to 10 hours,(tour_type != 'escort') & (duration > 7) & (duration < 11),coef_duration_constants_8_to_10_hours -util_duration_constants_11_to_13_hours,Duration Constants -- 11 to 13 hours,(tour_type != 'escort') & (duration > 10) & (duration < 14),coef_duration_constants_11_to_13_hours -util_duration_constants_14_to_18_hours,Duration Constants -- 14 to 18 hours,(tour_type != 'escort') & (duration > 13) & (duration < 19),coef_duration_constants_14_to_18_hours -util_escort_tour_departure_constants_early,Escort Tour Departure Constants -- Early (up to 5),(tour_type == 'escort') & (start < 6),coef_escort_tour_departure_constants_early -util_escort_tour_departure_constants_am_peak_1,Escort Tour Departure Constants -- AM peak 1 (6),(tour_type == 'escort') & (start == 6),coef_escort_tour_departure_constants_am_peak_1 -util_escort_tour_departure_constants_am_peak_2,Escort Tour Departure Constants -- AM peak 2 (7),(tour_type == 'escort') & (start == 7),coef_escort_tour_departure_constants_am_peak_2 -util_escort_tour_departure_constants_am_peak_3,Escort Tour Departure Constants -- AM peak 3 (8),(tour_type == 'escort') & (start == 8),coef_escort_tour_departure_constants_am_peak_3 -util_escort_tour_departure_constants_am_peak_4,Escort Tour Departure Constants -- AM peak 4 (9),(tour_type == 'escort') & (start == 9),coef_escort_tour_departure_constants_am_peak_4 -util_escort_tour_departure_constants_midday_1,Escort Tour Departure Constants -- Midday 1 (10 to 12),(tour_type == 'escort') & (start > 9) & (start < 13),coef_escort_tour_departure_constants_midday_1 -util_escort_tour_departure_constants_midday_2,Escort Tour Departure Constants -- Midday 2 (13 to 15),(tour_type == 'escort') & (start > 12) & (start < 16),coef_escort_tour_departure_constants_midday_2 -util_escort_tour_departure_constants_pm_peak,Escort Tour Departure Constants -- PM peak (16 to 18),(tour_type == 'escort') & (start > 15) & (start < 19),coef_escort_tour_departure_constants_pm_peak -util_escort_tour_departure_constants_evening,Escort Tour Departure Constants -- Evening (19 to 21),(tour_type == 'escort') & (start > 18) & (start < 22),coef_escort_tour_departure_constants_evening -util_escort_tour_departure_constants_late,Escort Tour Departure Constants -- Late (22 and later),(tour_type == 'escort') & (start > 21),coef_escort_tour_departure_constants_late -util_escort_tour_arrival_constants_early,Escort Tour Arrival Constants -- Early (up to 6),(tour_type == 'escort') & (end < 7),coef_escort_tour_arrival_constants_early -util_escort_tour_arrival_constants_am_peak,Escort Tour Arrival Constants -- AM peak (7 to 9),(tour_type == 'escort') & (end > 6) & (end < 10),coef_escort_tour_arrival_constants_am_peak -util_escort_tour_arrival_constants_midday_1,Escort Tour Arrival Constants -- Midday 1 (10 to 12),(tour_type == 'escort') & (end > 9) & (end < 13),coef_escort_tour_arrival_constants_midday_1 -util_escort_tour_arrival_constants_midday_2,Escort Tour Arrival Constants -- Midday 2 (13 to 14),(tour_type == 'escort') & (end > 12) & (end < 15),coef_escort_tour_arrival_constants_midday_2 -util_escort_tour_arrival_constants_pm_peak_1,Escort Tour Arrival Constants -- PM peak 1 (15),(tour_type == 'escort') & (end == 15),coef_escort_tour_arrival_constants_pm_peak_1 -util_escort_tour_arrival_constants_pm_peak_2,Escort Tour Arrival Constants -- PM peak 2 (16),(tour_type == 'escort') & (end == 16),coef_escort_tour_arrival_constants_pm_peak_2 -util_escort_tour_arrival_constants_pm_peak_3,Escort Tour Arrival Constants -- PM peak 3 (17),(tour_type == 'escort') & (end == 17),coef_escort_tour_arrival_constants_pm_peak_3 -util_escort_tour_arrival_constants_pm_peak_4,Escort Tour Arrival Constants -- PM peak 4 (18),(tour_type == 'escort') & (end == 18),coef_escort_tour_arrival_constants_pm_peak_4 -util_escort_tour_arrival_constants_evening,Escort Tour Arrival Constants -- Evening (19 to 21),(tour_type == 'escort') & (end > 18) & (end < 22),coef_escort_tour_arrival_constants_evening -util_escort_tour_arrival_constants_late,Escort Tour Arrival Constants -- Late (22 and later),(tour_type == 'escort') & (end > 21),coef_escort_tour_arrival_constants_late -util_escort_tour_duration_constants_0_to_1_hours,Escort Tour Duration Constants -- 0 to 1 hours,(tour_type == 'escort') & (duration < 2),coef_escort_tour_duration_constants_0_to_1_hours -util_escort_tour_duration_constants_2_to_3_hours,Escort Tour Duration Constants -- 2 to 3 hours,(tour_type == 'escort') & (duration > 1) & (duration < 4),coef_escort_tour_duration_constants_2_to_3_hours -util_escort_tour_duration_constants_4_to_5_hours,Escort Tour Duration Constants -- 4 to 5 hours,(tour_type == 'escort') & (duration > 3) & (duration < 6),coef_escort_tour_duration_constants_4_to_5_hours -util_escort_tour_duration_constants_6_to_7_hours,Escort Tour Duration Constants -- 6 to 7 hours,(tour_type == 'escort') & (duration > 5) & (duration < 8),coef_escort_tour_duration_constants_6_to_7_hours -util_escort_tour_duration_constants_8_to_10_hours,Escort Tour Duration Constants -- 8 to 10 hours,(tour_type == 'escort') & (duration > 7) & (duration < 11),coef_escort_tour_duration_constants_8_to_10_hours -util_escort_tour_duration_constants_11_to_13_hours,Escort Tour Duration Constants -- 11 to 13 hours,(tour_type == 'escort') & (duration > 10) & (duration < 14),coef_escort_tour_duration_constants_11_to_13_hours -util_escort_tour_duration_constants_14_to_18_hours,Escort Tour Duration Constants -- 14 to 18 hours,(tour_type == 'escort') & (duration > 13) & (duration < 19),coef_escort_tour_duration_constants_14_to_18_hours +util_departure_constants_early,Departure Constants -- Early (up to 5),@(_is_not_escort) & (df.start < 6),coef_departure_constants_early +util_departure_constants_am_peak_1,Departure Constants -- AM peak 1 (6),@(_is_not_escort) & (df.start == 6),coef_departure_constants_am_peak_1 +util_departure_constants_am_peak_2,Departure Constants -- AM peak 2 (7),@(_is_not_escort) & (df.start == 7),coef_departure_constants_am_peak_2 +util_departure_constants_am_peak_3,Departure Constants -- AM peak 3 (8),@(_is_not_escort) & (df.start == 8),coef_departure_constants_am_peak_3 +util_departure_constants_am_peak_4,Departure Constants -- AM peak 4 (9),@(_is_not_escort) & (df.start == 9),coef_departure_constants_am_peak_4 +util_departure_constants_midday_1,Departure Constants -- Midday 1 (10 to 12),@(_is_not_escort) & (df.start > 9) & (df.start < 13),coef_departure_constants_midday_1 +util_departure_constants_midday_2,Departure Constants -- Midday 2 (13 to 15),@(_is_not_escort) & (df.start > 12) & (df.start < 16),coef_departure_constants_midday_2 +util_departure_constants_pm_peak,Departure Constants -- PM peak (16 to 18),@(_is_not_escort) & (df.start > 15) & (df.start < 19),coef_departure_constants_pm_peak +util_departure_constants_evening,Departure Constants -- Evening (19 to 21),@(_is_not_escort) & (df.start > 18) & (df.start < 22),coef_departure_constants_evening +util_departure_constants_late,Departure Constants -- Late (22 and later),@(_is_not_escort) & (df.start > 21),coef_departure_constants_late +util_arrival_constants_early,Arrival Constants -- Early (up to 6),@(_is_not_escort) & (df.end < 7),coef_arrival_constants_early +util_arrival_constants_am_peak,Arrival Constants -- AM peak (7 to 9),@(_is_not_escort) & (df.end > 6) & (df.end < 10),coef_arrival_constants_am_peak +util_arrival_constants_midday_1,Arrival Constants -- Midday 1 (10 to 12),@(_is_not_escort) & (df.end > 9) & (df.end < 13),coef_arrival_constants_midday_1 +util_arrival_constants_midday_2,Arrival Constants -- Midday 2 (13 to 14),@(_is_not_escort) & (df.end > 12) & (df.end < 15),coef_arrival_constants_midday_2 +util_arrival_constants_pm_peak_1,Arrival Constants -- PM peak 1 (15),@(_is_not_escort) & (df.end == 15),coef_arrival_constants_pm_peak_1 +util_arrival_constants_pm_peak_2,Arrival Constants -- PM peak 2 (16),@(_is_not_escort) & (df.end == 16),coef_arrival_constants_pm_peak_2 +util_arrival_constants_pm_peak_3,Arrival Constants -- PM peak 3 (17),@(_is_not_escort) & (df.end == 17),coef_arrival_constants_pm_peak_3 +util_arrival_constants_pm_peak_4,Arrival Constants -- PM peak 4 (18),@(_is_not_escort) & (df.end == 18),coef_arrival_constants_pm_peak_4 +util_arrival_constants_evening,Arrival Constants -- Evening (19 to 21),@(_is_not_escort) & (df.end > 18) & (df.end < 22),coef_arrival_constants_evening +util_arrival_constants_late,Arrival Constants -- Late (22 and later),@(_is_not_escort) & (df.end > 21),coef_arrival_constants_late +util_duration_constants_0_to_1_hours,Duration Constants -- 0 to 1 hours,@(_is_not_escort) & (df.duration < 2),coef_duration_constants_0_to_1_hours +util_duration_constants_2_to_3_hours,Duration Constants -- 2 to 3 hours,@(_is_not_escort) & (df.duration > 1) & (df.duration < 4),coef_duration_constants_2_to_3_hours +util_duration_constants_4_to_5_hours,Duration Constants -- 4 to 5 hours,@(_is_not_escort) & (df.duration > 3) & (df.duration < 6),coef_duration_constants_4_to_5_hours +util_duration_constants_6_to_7_hours,Duration Constants -- 6 to 7 hours,@(_is_not_escort) & (df.duration > 5) & (df.duration < 8),coef_duration_constants_6_to_7_hours +util_duration_constants_8_to_10_hours,Duration Constants -- 8 to 10 hours,@(_is_not_escort) & (df.duration > 7) & (df.duration < 11),coef_duration_constants_8_to_10_hours +util_duration_constants_11_to_13_hours,Duration Constants -- 11 to 13 hours,@(_is_not_escort) & (df.duration > 10) & (df.duration < 14),coef_duration_constants_11_to_13_hours +util_duration_constants_14_to_18_hours,Duration Constants -- 14 to 18 hours,@(_is_not_escort) & (df.duration > 13) & (df.duration < 19),coef_duration_constants_14_to_18_hours +util_escort_tour_departure_constants_early,Escort Tour Departure Constants -- Early (up to 5),@(_is_escort) & (df.start < 6),coef_escort_tour_departure_constants_early +util_escort_tour_departure_constants_am_peak_1,Escort Tour Departure Constants -- AM peak 1 (6),@(_is_escort) & (df.start == 6),coef_escort_tour_departure_constants_am_peak_1 +util_escort_tour_departure_constants_am_peak_2,Escort Tour Departure Constants -- AM peak 2 (7),@(_is_escort) & (df.start == 7),coef_escort_tour_departure_constants_am_peak_2 +util_escort_tour_departure_constants_am_peak_3,Escort Tour Departure Constants -- AM peak 3 (8),@(_is_escort) & (df.start == 8),coef_escort_tour_departure_constants_am_peak_3 +util_escort_tour_departure_constants_am_peak_4,Escort Tour Departure Constants -- AM peak 4 (9),@(_is_escort) & (df.start == 9),coef_escort_tour_departure_constants_am_peak_4 +util_escort_tour_departure_constants_midday_1,Escort Tour Departure Constants -- Midday 1 (10 to 12),@(_is_escort) & (df.start > 9) & (df.start < 13),coef_escort_tour_departure_constants_midday_1 +util_escort_tour_departure_constants_midday_2,Escort Tour Departure Constants -- Midday 2 (13 to 15),@(_is_escort) & (df.start > 12) & (df.start < 16),coef_escort_tour_departure_constants_midday_2 +util_escort_tour_departure_constants_pm_peak,Escort Tour Departure Constants -- PM peak (16 to 18),@(_is_escort) & (df.start > 15) & (df.start < 19),coef_escort_tour_departure_constants_pm_peak +util_escort_tour_departure_constants_evening,Escort Tour Departure Constants -- Evening (19 to 21),@(_is_escort) & (df.start > 18) & (df.start < 22),coef_escort_tour_departure_constants_evening +util_escort_tour_departure_constants_late,Escort Tour Departure Constants -- Late (22 and later),@(_is_escort) & (df.start > 21),coef_escort_tour_departure_constants_late +util_escort_tour_arrival_constants_early,Escort Tour Arrival Constants -- Early (up to 6),@(_is_escort) & (df.end < 7),coef_escort_tour_arrival_constants_early +util_escort_tour_arrival_constants_am_peak,Escort Tour Arrival Constants -- AM peak (7 to 9),@(_is_escort) & (df.end > 6) & (df.end < 10),coef_escort_tour_arrival_constants_am_peak +util_escort_tour_arrival_constants_midday_1,Escort Tour Arrival Constants -- Midday 1 (10 to 12),@(_is_escort) & (df.end > 9) & (df.end < 13),coef_escort_tour_arrival_constants_midday_1 +util_escort_tour_arrival_constants_midday_2,Escort Tour Arrival Constants -- Midday 2 (13 to 14),@(_is_escort) & (df.end > 12) & (df.end < 15),coef_escort_tour_arrival_constants_midday_2 +util_escort_tour_arrival_constants_pm_peak_1,Escort Tour Arrival Constants -- PM peak 1 (15),@(_is_escort) & (df.end == 15),coef_escort_tour_arrival_constants_pm_peak_1 +util_escort_tour_arrival_constants_pm_peak_2,Escort Tour Arrival Constants -- PM peak 2 (16),@(_is_escort) & (df.end == 16),coef_escort_tour_arrival_constants_pm_peak_2 +util_escort_tour_arrival_constants_pm_peak_3,Escort Tour Arrival Constants -- PM peak 3 (17),@(_is_escort) & (df.end == 17),coef_escort_tour_arrival_constants_pm_peak_3 +util_escort_tour_arrival_constants_pm_peak_4,Escort Tour Arrival Constants -- PM peak 4 (18),@(_is_escort) & (df.end == 18),coef_escort_tour_arrival_constants_pm_peak_4 +util_escort_tour_arrival_constants_evening,Escort Tour Arrival Constants -- Evening (19 to 21),@(_is_escort) & (df.end > 18) & (df.end < 22),coef_escort_tour_arrival_constants_evening +util_escort_tour_arrival_constants_late,Escort Tour Arrival Constants -- Late (22 and later),@(_is_escort) & (df.end > 21),coef_escort_tour_arrival_constants_late +util_escort_tour_duration_constants_0_to_1_hours,Escort Tour Duration Constants -- 0 to 1 hours,@(_is_escort) & (df.duration < 2),coef_escort_tour_duration_constants_0_to_1_hours +util_escort_tour_duration_constants_2_to_3_hours,Escort Tour Duration Constants -- 2 to 3 hours,@(_is_escort) & (df.duration > 1) & (df.duration < 4),coef_escort_tour_duration_constants_2_to_3_hours +util_escort_tour_duration_constants_4_to_5_hours,Escort Tour Duration Constants -- 4 to 5 hours,@(_is_escort) & (df.duration > 3) & (df.duration < 6),coef_escort_tour_duration_constants_4_to_5_hours +util_escort_tour_duration_constants_6_to_7_hours,Escort Tour Duration Constants -- 6 to 7 hours,@(_is_escort) & (df.duration > 5) & (df.duration < 8),coef_escort_tour_duration_constants_6_to_7_hours +util_escort_tour_duration_constants_8_to_10_hours,Escort Tour Duration Constants -- 8 to 10 hours,@(_is_escort) & (df.duration > 7) & (df.duration < 11),coef_escort_tour_duration_constants_8_to_10_hours +util_escort_tour_duration_constants_11_to_13_hours,Escort Tour Duration Constants -- 11 to 13 hours,@(_is_escort) & (df.duration > 10) & (df.duration < 14),coef_escort_tour_duration_constants_11_to_13_hours +util_escort_tour_duration_constants_14_to_18_hours,Escort Tour Duration Constants -- 14 to 18 hours,@(_is_escort) & (df.duration > 13) & (df.duration < 19),coef_escort_tour_duration_constants_14_to_18_hours diff --git a/activitysim/examples/prototype_mtc/configs/tour_scheduling_school.csv b/activitysim/examples/prototype_mtc/configs/tour_scheduling_school.csv index 4c34c1edce..60dfc01704 100644 --- a/activitysim/examples/prototype_mtc/configs/tour_scheduling_school.csv +++ b/activitysim/examples/prototype_mtc/configs/tour_scheduling_school.csv @@ -21,8 +21,8 @@ util_school_plus_work_tours_by_worker_lt_6_hours,School+work tours by worker- du util_mode_choice_logsum,Mode Choice Logsum,mode_choice_logsum,coef_mode_choice_logsum #,,, #,,, FIXME - use temps as timetable ops can be very time-consuming -util_dummy_adjacent_before,,"_adjacent_window_before@tt.adjacent_window_before(df.person_id, df.start)",coef_dummy -util_dummy_adjacent_after,,"_adjacent_window_after@tt.adjacent_window_after(df.person_id, df.end)",coef_dummy +util_dummy_adjacent_before,,"_adjacent_window_before@tt.adjacent_window_before(df.person_id, df.start)>0",coef_dummy +util_dummy_adjacent_after,,"_adjacent_window_after@tt.adjacent_window_after(df.person_id, df.end)>0",coef_dummy util_previous_tour_ends_this_departure_hour,Previously-scheduled tour ends in this departure hour,"@tt.previous_tour_ends(df.person_id, df.start)",coef_previous_tour_ends_this_departure_hour util_previous_tour_begins_this_arrival_hour,Previously-scheduled tour begins in this arrival hour,"@tt.previous_tour_begins(df.person_id, df.end)",coef_previous_tour_begins_this_arrival_hour coef_adjacent_window_exists_before_this_departure_hour_first_tour_interaction,Adjacent window exists before this departure hour - first tour interaction,"@(df.tour_count>1) & (df.tour_num == 1) & _adjacent_window_before",coef_adjacent_window_exists_before_this_departure_hour_first_tour_interaction diff --git a/activitysim/examples/prototype_mtc/configs/tour_scheduling_work.csv b/activitysim/examples/prototype_mtc/configs/tour_scheduling_work.csv index 7aa6a363a9..2abe8801de 100644 --- a/activitysim/examples/prototype_mtc/configs/tour_scheduling_work.csv +++ b/activitysim/examples/prototype_mtc/configs/tour_scheduling_work.csv @@ -29,8 +29,8 @@ util_tours_by_student_duration_lt_8_hrs,School+work tours by student- duration<8 util_mode_choice_logsum,Mode Choice Logsum,mode_choice_logsum,coef_mode_choice_logsum #,,, #,FIXME - use temps _adjacent_window_before and _adjacent_window_after because timetable ops can be very time-consuming,, -util_dummy_adjacent_before,local temp variable,"_adjacent_window_before@tt.adjacent_window_before(df.person_id, df.start)",coef_dummy -util_dummy_adjacent_after,local temp variable,"_adjacent_window_after@tt.adjacent_window_after(df.person_id, df.end)",coef_dummy +util_dummy_adjacent_before,local temp variable,"_adjacent_window_before@tt.adjacent_window_before(df.person_id, df.start)>0",coef_dummy +util_dummy_adjacent_after,local temp variable,"_adjacent_window_after@tt.adjacent_window_after(df.person_id, df.end)>0",coef_dummy util_previous_tour_ends_this_departure_hour,Previously-scheduled tour ends in this departure hour,"@tt.previous_tour_ends(df.person_id, df.start)",coef_previously_scheduled_tour_ends_in_this_departure_hour util_previously_scheduled_tour_begins_in_this_arrival_hour,Previously-scheduled tour begins in this arrival hour,"@tt.previous_tour_begins(df.person_id, df.end)",coef_previously_scheduled_tour_begins_in_this_arrival_hour util_adjacent_window_exists_before_this_departure_hour_first_tour_interaction,Adjacent window exists before this departure hour - first tour interaction,"@(df.tour_count>1) & (df.tour_num == 1) & _adjacent_window_before",coef_adjacent_window_exists_before_this_departure_hour_first_tour_interaction diff --git a/activitysim/examples/prototype_mtc/configs/trip_destination.csv b/activitysim/examples/prototype_mtc/configs/trip_destination.csv index b12246c13f..5ebf614d32 100644 --- a/activitysim/examples/prototype_mtc/configs/trip_destination.csv +++ b/activitysim/examples/prototype_mtc/configs/trip_destination.csv @@ -1,8 +1,8 @@ Label,Description,Expression,work,univ,school,escort,shopping,eatout,othmaint,social,othdiscr,atwork local_dist_od,,_od_DIST@od_skims['DIST'],1,1,1,1,1,1,1,1,1,1 local_dist_dp,,_dp_DIST@dp_skims['DIST'],1,1,1,1,1,1,1,1,1,1 -util_size_term,size term,"@np.log1p(size_terms.get(df.dest_taz, df.purpose))",coef_one,coef_one,coef_one,coef_one,coef_one,coef_one,coef_one,coef_one,coef_one,coef_one -util_no_attractions,no attractions,"@size_terms.get(df.dest_taz, df.purpose) == 0",coef_UNAVAILABLE,coef_UNAVAILABLE,coef_UNAVAILABLE,coef_UNAVAILABLE,coef_UNAVAILABLE,coef_UNAVAILABLE,coef_UNAVAILABLE,coef_UNAVAILABLE,coef_UNAVAILABLE,coef_UNAVAILABLE +util_size_term,size term,"@np.log1p(size_terms.get(df.dest_taz, df.purpose)) # sharrow: np.log1p(size_terms['arry'])",coef_one,coef_one,coef_one,coef_one,coef_one,coef_one,coef_one,coef_one,coef_one,coef_one +util_no_attractions,no attractions,"@size_terms.get(df.dest_taz, df.purpose) == 0 # sharrow: size_terms['arry'] == 0",coef_UNAVAILABLE,coef_UNAVAILABLE,coef_UNAVAILABLE,coef_UNAVAILABLE,coef_UNAVAILABLE,coef_UNAVAILABLE,coef_UNAVAILABLE,coef_UNAVAILABLE,coef_UNAVAILABLE,coef_UNAVAILABLE util_stop_zone_CDB_are_type,#stop zone CBD area type,"@reindex(land_use.area_type, df.dest_taz) < setting('cbd_threshold')",,,,,,,,,, util_distance_inbound,distance (calibration adjustment individual - inbound),@(~df.is_joint & ~df.outbound) * (_od_DIST + _dp_DIST),coef_util_distance_work_outbound,coef_util_distance_univ,coef_util_distance_school,coef_util_distance_escort,coef_util_distance_shopping,coef_util_distance_eatout,coef_util_distance_othmaint,coef_util_distance_social,coef_util_distance_othdiscr,coef_util_distance_atwork util_distance_outbound,distance (calibration adjustment individual - outbound),@(~df.is_joint & df.outbound) * (_od_DIST + _dp_DIST),coef_util_distance_work_inbound,coef_util_distance_univ,coef_util_distance_school,coef_util_distance_escort,coef_util_distance_shopping,coef_util_distance_eatout,coef_util_distance_othmaint,coef_util_distance_social,coef_util_distance_othdiscr,coef_util_distance_atwork diff --git a/activitysim/examples/prototype_mtc/configs/trip_destination_annotate_trips_preprocessor.csv b/activitysim/examples/prototype_mtc/configs/trip_destination_annotate_trips_preprocessor.csv index 1a1afb0748..b99cbc98f9 100644 --- a/activitysim/examples/prototype_mtc/configs/trip_destination_annotate_trips_preprocessor.csv +++ b/activitysim/examples/prototype_mtc/configs/trip_destination_annotate_trips_preprocessor.csv @@ -1,5 +1,4 @@ Description,Target,Expression -#,, ,tour_mode,"reindex(tours.tour_mode, df.tour_id)" ,_tod,"np.where(df.outbound,reindex_i(tours.start, df.tour_id),reindex_i(tours.end, df.tour_id))" ,trip_period,network_los.skim_time_period_label(_tod) @@ -7,4 +6,7 @@ Description,Target,Expression #,,not needed as school is not chosen as an intermediate trip destination #,_grade_school,"(df.primary_purpose == 'school') & reindex(persons.is_gradeschool, df.person_id)" #,size_segment,"df.primary_purpose.where(df.primary_purpose != 'school', np.where(_grade_school,'gradeschool', 'highschool'))" -,tour_leg_dest,"np.where(df.outbound,reindex(tours.destination, df.tour_id), reindex(tours.origin, df.tour_id))" \ No newline at end of file +,tour_leg_dest,"np.where(df.outbound,reindex(tours.destination, df.tour_id), reindex(tours.origin, df.tour_id))" +,purpose_index_num,"size_terms.get_cols(df.purpose)" +,tour_mode_is_walk,"reindex(tours.tour_mode, df.tour_id)=='WALK'" +,tour_mode_is_bike,"reindex(tours.tour_mode, df.tour_id)=='BIKE'" diff --git a/activitysim/examples/prototype_mtc/configs/trip_destination_sample.csv b/activitysim/examples/prototype_mtc/configs/trip_destination_sample.csv index 857fb44c03..c168420ec6 100644 --- a/activitysim/examples/prototype_mtc/configs/trip_destination_sample.csv +++ b/activitysim/examples/prototype_mtc/configs/trip_destination_sample.csv @@ -1,13 +1,13 @@ Description,Expression,work,univ,school,escort,shopping,eatout,othmaint,social,othdiscr,atwork ,_od_DIST@od_skims['DIST'],1,1,1,1,1,1,1,1,1,1 ,_dp_DIST@dp_skims['DIST'],1,1,1,1,1,1,1,1,1,1 -Not available if walk tour not within walking distance,@(df.tour_mode=='WALK') & (od_skims['DISTWALK'] > max_walk_distance),-999,-999,-999,-999,-999,-999,-999,-999,-999,-999 -Not available if walk tour not within walking distance,@(df.tour_mode=='WALK') & (dp_skims['DISTWALK'] > max_walk_distance),-999,-999,-999,-999,-999,-999,-999,-999,-999,-999 -Not available if bike tour not within biking distance,@(df.tour_mode=='BIKE') & (od_skims['DISTBIKE'] > max_bike_distance),-999,-999,-999,-999,-999,-999,-999,-999,-999,-999 -Not available if bike tour not within biking distance,@(df.tour_mode=='BIKE') & (dp_skims['DISTBIKE'] > max_bike_distance),-999,-999,-999,-999,-999,-999,-999,-999,-999,-999 +Not available if walk tour not within walking distance,@(df.tour_mode_is_walk) & (od_skims['DISTWALK'] > max_walk_distance),-999,-999,-999,-999,-999,-999,-999,-999,-999,-999 +Not available if walk tour not within walking distance,@(df.tour_mode_is_walk) & (dp_skims['DISTWALK'] > max_walk_distance),-999,-999,-999,-999,-999,-999,-999,-999,-999,-999 +Not available if bike tour not within biking distance,@(df.tour_mode_is_bike) & (od_skims['DISTBIKE'] > max_bike_distance),-999,-999,-999,-999,-999,-999,-999,-999,-999,-999 +Not available if bike tour not within biking distance,@(df.tour_mode_is_bike) & (dp_skims['DISTBIKE'] > max_bike_distance),-999,-999,-999,-999,-999,-999,-999,-999,-999,-999 #If transit tour is not in walk sub-zone it must be walkable,,,,,,,,,,, -size term,"@np.log1p(size_terms.get(df.dest_taz, df.purpose))",1,1,1,1,1,1,1,1,1,1 -no attractions,"@size_terms.get(df.dest_taz, df.purpose) == 0",-999,-999,-999,-999,-999,-999,-999,-999,-999,-999 +size term,"@np.log1p(size_terms.get(df.dest_taz, df.purpose)) # sharrow: np.log1p(size_terms['arry'])",1,1,1,1,1,1,1,1,1,1 +no attractions,"@size_terms.get(df.dest_taz, df.purpose) == 0 # sharrow: size_terms['arry'] == 0",-999,-999,-999,-999,-999,-999,-999,-999,-999,-999 #stop zone CBD area type,"@reindex(land_use.area_type, df.dest_taz) < setting('cbd_threshold')",,,,,,,,,, distance (calibration adjustment individual - inbound),@(~df.is_joint & ~df.outbound) * (_od_DIST + _dp_DIST),-0.04972591574229,-0.0613,-0.1056,-0.1491,-0.1192,-0.1029,-0.0962,-0.1329,-0.126172224,-0.122334597 distance (calibration adjustment individual - outbound),@(~df.is_joint & df.outbound) * (_od_DIST + _dp_DIST),0.147813278663948,-0.0613,-0.1056,-0.1491,-0.1192,-0.1029,-0.0962,-0.1329,-0.126172224,-0.122334597 diff --git a/activitysim/examples/prototype_mtc/configs/trip_mode_choice.csv b/activitysim/examples/prototype_mtc/configs/trip_mode_choice.csv index 3ac15254ba..590f470733 100644 --- a/activitysim/examples/prototype_mtc/configs/trip_mode_choice.csv +++ b/activitysim/examples/prototype_mtc/configs/trip_mode_choice.csv @@ -330,18 +330,18 @@ util_tour_mode_is_walk_transit,Walk to Transit tour mode availability,tour_mode_ util_tour_mode_is_drive_transit,Drive to Transit tour modes availability,tour_mode_is_drive_transit,-999,-999,-999,-999,-999,-999,-999,-999,-999,-999,-999,-999,-999,,,,,,,, util_tour_mode_is_ride_hail,Ride hail tour modes availability,tour_mode_is_ride_hail,-999,-999,,,,,,-999,,,,,,-999,-999,-999,-999,-999,,, ,#indiv tour ASCs,,,,,,,,,,,,,,,,,,,,,, -util_Drive_Alone_tour_mode_ASC_shared_ride_2_df_is_indiv,Drive Alone tour mode ASC -- shared ride 2,@(df.is_indiv & df.i_tour_mode.isin(I_SOV_MODES)),,,sov_ASC_sr2,sov_ASC_sr2,,,,,,,,,,,,,,,,, -util_Drive_Alone_tour_mode_ASC_shared_ride_3_plus,Drive Alone tour mode ASC -- shared ride 3+,@(df.is_indiv & df.i_tour_mode.isin(I_SOV_MODES)),,,,,sov_ASC_sr3p,sov_ASC_sr3p,,,,,,,,,,,,,,, -util_Drive_Alone_tour_mode_ASC_walk,Drive Alone tour mode ASC -- walk,@(df.is_indiv & df.i_tour_mode.isin(I_SOV_MODES)),,,,,,,sov_ASC_walk,,,,,,,,,,,,,, -util_Drive_Alone_tour_mode_ASC_ride_hail,Drive Alone tour mode ASC -- ride hail,@(df.is_indiv & df.i_tour_mode.isin(I_SOV_MODES)),,,,,,,,,,,,,,,,,,,sov_ASC_rh,sov_ASC_rh,sov_ASC_rh -util_Shared_Ride_2_tour_mode_ASC_shared_ride_2,Shared Ride 2 tour mode ASC -- shared ride 2,@(df.is_indiv & df.i_tour_mode.isin(I_SR2_MODES)),,,sr2_ASC_sr2,sr2_ASC_sr2,,,,,,,,,,,,,,,,, -util_Shared_Ride_2_tour_mode_ASC_shared_ride_3_plus,Shared Ride 2 tour mode ASC -- shared ride 3+,@(df.is_indiv & df.i_tour_mode.isin(I_SR2_MODES)),,,,,sr2_ASC_sr3p,sr2_ASC_sr3p,,,,,,,,,,,,,,, -util_Shared_Ride_2_tour_mode_ASC_walk,Shared Ride 2 tour mode ASC -- walk,@(df.is_indiv & df.i_tour_mode.isin(I_SR2_MODES)),,,,,,,sr2_ASC_walk,,,,,,,,,,,,,, -util_Shared_Ride_2_tour_mode_ASC_ride_hail,Shared Ride 2 tour mode ASC -- ride hail,@(df.is_indiv & df.i_tour_mode.isin(I_SR2_MODES)),,,,,,,,,,,,,,,,,,,sr2_ASC_rh,sr2_ASC_rh,sr2_ASC_rh -util_Shared_Ride_3_tour_mode_ASC_shared_ride_2,Shared Ride 3+ tour mode ASC -- shared ride 2,@(df.is_indiv & df.i_tour_mode.isin(I_SR3P_MODES)),,,sr3p_ASC_sr2,sr3p_ASC_sr2,,,,,,,,,,,,,,,,, -util_Shared_Ride_3_tour_mode_ASC_shared_ride_3_plus,Shared Ride 3+ tour mode ASC -- shared ride 3+,@(df.is_indiv & df.i_tour_mode.isin(I_SR3P_MODES)),,,,,sr3p_ASC_sr3p,sr3p_ASC_sr3p,,,,,,,,,,,,,,, -util_Shared_Ride_3_tour_mode_ASC_walk,Shared Ride 3+ tour mode ASC -- walk,@(df.is_indiv & df.i_tour_mode.isin(I_SR3P_MODES)),,,,,,,sr3p_ASC_walk,,,,,,,,,,,,,, -util_Shared_Ride_3_tour_mode_ASC_ride_hail,Shared Ride 3+ tour mode ASC -- ride hail,@(df.is_indiv & df.i_tour_mode.isin(I_SR3P_MODES)),,,,,,,,,,,,,,,,,,,sr3p_ASC_rh,sr3p_ASC_rh,sr3p_ASC_rh +util_Drive_Alone_tour_mode_ASC_shared_ride_2_df_is_indiv,Drive Alone tour mode ASC -- shared ride 2,@(df.is_indiv & df.tour_mode_is_SOV),,,sov_ASC_sr2,sov_ASC_sr2,,,,,,,,,,,,,,,,, +util_Drive_Alone_tour_mode_ASC_shared_ride_3_plus,Drive Alone tour mode ASC -- shared ride 3+,@(df.is_indiv & df.tour_mode_is_SOV),,,,,sov_ASC_sr3p,sov_ASC_sr3p,,,,,,,,,,,,,,, +util_Drive_Alone_tour_mode_ASC_walk,Drive Alone tour mode ASC -- walk,@(df.is_indiv & df.tour_mode_is_SOV),,,,,,,sov_ASC_walk,,,,,,,,,,,,,, +util_Drive_Alone_tour_mode_ASC_ride_hail,Drive Alone tour mode ASC -- ride hail,@(df.is_indiv & df.tour_mode_is_SOV),,,,,,,,,,,,,,,,,,,sov_ASC_rh,sov_ASC_rh,sov_ASC_rh +util_Shared_Ride_2_tour_mode_ASC_shared_ride_2,Shared Ride 2 tour mode ASC -- shared ride 2,@(df.is_indiv & df.tour_mode_is_SR2),,,sr2_ASC_sr2,sr2_ASC_sr2,,,,,,,,,,,,,,,,, +util_Shared_Ride_2_tour_mode_ASC_shared_ride_3_plus,Shared Ride 2 tour mode ASC -- shared ride 3+,@(df.is_indiv & df.tour_mode_is_SR2),,,,,sr2_ASC_sr3p,sr2_ASC_sr3p,,,,,,,,,,,,,,, +util_Shared_Ride_2_tour_mode_ASC_walk,Shared Ride 2 tour mode ASC -- walk,@(df.is_indiv & df.tour_mode_is_SR2),,,,,,,sr2_ASC_walk,,,,,,,,,,,,,, +util_Shared_Ride_2_tour_mode_ASC_ride_hail,Shared Ride 2 tour mode ASC -- ride hail,@(df.is_indiv & df.tour_mode_is_SR2),,,,,,,,,,,,,,,,,,,sr2_ASC_rh,sr2_ASC_rh,sr2_ASC_rh +util_Shared_Ride_3_tour_mode_ASC_shared_ride_2,Shared Ride 3+ tour mode ASC -- shared ride 2,@(df.is_indiv & df.tour_mode_is_SR3P),,,sr3p_ASC_sr2,sr3p_ASC_sr2,,,,,,,,,,,,,,,,, +util_Shared_Ride_3_tour_mode_ASC_shared_ride_3_plus,Shared Ride 3+ tour mode ASC -- shared ride 3+,@(df.is_indiv & df.tour_mode_is_SR3P),,,,,sr3p_ASC_sr3p,sr3p_ASC_sr3p,,,,,,,,,,,,,,, +util_Shared_Ride_3_tour_mode_ASC_walk,Shared Ride 3+ tour mode ASC -- walk,@(df.is_indiv & df.tour_mode_is_SR3P),,,,,,,sr3p_ASC_walk,,,,,,,,,,,,,, +util_Shared_Ride_3_tour_mode_ASC_ride_hail,Shared Ride 3+ tour mode ASC -- ride hail,@(df.is_indiv & df.tour_mode_is_SR3P),,,,,,,,,,,,,,,,,,,sr3p_ASC_rh,sr3p_ASC_rh,sr3p_ASC_rh util_Walk_tour_mode_ASC_ride_hail,Walk tour mode ASC -- ride hail,@df.is_indiv & (df.i_tour_mode == I_WALK_MODE),,,,,,,,,,,,,,,,,,,walk_ASC_rh,walk_ASC_rh,walk_ASC_rh util_Bike_tour_mode_ASC_walk,Bike tour mode ASC -- walk,@df.is_indiv & (df.i_tour_mode == I_BIKE_MODE),,,,,,,bike_ASC_walk,,,,,,,,,,,,,, util_Bike_tour_mode_ASC_ride_hail,Bike tour mode ASC -- ride hail,@df.is_indiv & (df.i_tour_mode == I_BIKE_MODE),,,,,,,,,,,,,,,,,,,bike_ASC_rh,bike_ASC_rh,bike_ASC_rh @@ -364,15 +364,15 @@ util_Ride_Hail_tour_mode_ASC_shared_ride_2,Ride Hail tour mode ASC -- shared rid util_Ride_Hail_tour_mode_ASC_shared_ride_3_plus,Ride Hail tour mode ASC -- shared ride 3+,@(df.is_indiv & df.tour_mode_is_ride_hail),,,,,ride_hail_ASC_sr3p,ride_hail_ASC_sr3p,,,,,,,,,,,,,,, util_Ride_Hail_tour_mode_ASC_walk,Ride Hail tour mode ASC -- walk,@(df.is_indiv & df.tour_mode_is_ride_hail),,,,,,,ride_hail_ASC_walk,,,,,,,,,,,,,, util_Ride_Hail_tour_mode_ASC_walk_to_transit,Ride Hail tour mode ASC -- walk to transit,@(df.is_indiv & df.tour_mode_is_ride_hail),,,,,,,,,ride_hail_ASC_walk_transit,ride_hail_ASC_walk_transit,ride_hail_ASC_walk_transit,ride_hail_ASC_walk_transit,ride_hail_ASC_walk_transit,,,,,,,, -util_Ride_Hail_tour_mode_ASC_ride_hail_taxi,Ride Hail tour mode ASC -- ride hail,@(df.is_indiv & df.i_tour_mode.isin(I_RIDE_HAIL_MODES)),,,,,,,,,,,,,,,,,,,ride_hail_ASC_taxi,, -util_Ride_Hail_tour_mode_ASC_ride_hail_single,Ride Hail tour mode ASC -- ride hail,@(df.is_indiv & df.i_tour_mode.isin(I_RIDE_HAIL_MODES)),,,,,,,,,,,,,,,,,,,,ride_hail_ASC_tnc_single, -util_Ride_Hail_tour_mode_ASC_ride_hail_shared,Ride Hail tour mode ASC -- ride hail,@(df.is_indiv & df.i_tour_mode.isin(I_RIDE_HAIL_MODES)),,,,,,,,,,,,,,,,,,,,,ride_hail_ASC_tnc_shared +util_Ride_Hail_tour_mode_ASC_ride_hail_taxi,Ride Hail tour mode ASC -- ride hail,@(df.is_indiv & df.tour_mode_is_ride_hail),,,,,,,,,,,,,,,,,,,ride_hail_ASC_taxi,, +util_Ride_Hail_tour_mode_ASC_ride_hail_single,Ride Hail tour mode ASC -- ride hail,@(df.is_indiv & df.tour_mode_is_ride_hail),,,,,,,,,,,,,,,,,,,,ride_hail_ASC_tnc_single, +util_Ride_Hail_tour_mode_ASC_ride_hail_shared,Ride Hail tour mode ASC -- ride hail,@(df.is_indiv & df.tour_mode_is_ride_hail),,,,,,,,,,,,,,,,,,,,,ride_hail_ASC_tnc_shared #,joint tour ASCs,,,,,,,,,,,,,,,,,,,,,, -util_joint_auto_tour_mode_ASC_shared_ride_2,joint - auto tour mode ASC -- shared ride 2,@(df.is_joint & df.i_tour_mode.isin(I_AUTO_MODES)),,,joint_auto_ASC_sr2,joint_auto_ASC_sr2,,,,,,,,,,,,,,,,, -util_joint_auto_tour_mode_ASC_shared_ride_3_,joint - auto tour mode ASC -- shared ride 3+,@(df.is_joint & df.i_tour_mode.isin(I_AUTO_MODES)),,,,,joint_auto_ASC_sr3p,joint_auto_ASC_sr3p,,,,,,,,,,,,,,, -util_joint_auto_tour_mode_ASC_walk,joint - auto tour mode ASC -- walk,@(df.is_joint & df.i_tour_mode.isin(I_AUTO_MODES)),,,,,,,joint_auto_ASC_walk,,,,,,,,,,,,,, -util_joint_auto_tour_mode_ASC_ride_hail,joint - auto tour mode ASC -- ride hail,@(df.is_joint & df.i_tour_mode.isin(I_RIDE_HAIL_MODES)),,,,,,,,,,,,,,,,,,,joint_auto_ASC_rh,joint_auto_ASC_rh,joint_auto_ASC_rh -util_joint_Walk_tour_mode_ASC_ride_hail,joint - Walk tour mode ASC -- ride hail,@(df.is_joint & df.i_tour_mode.isin(I_RIDE_HAIL_MODES)),,,,,,,joint_walk_ASC_rh,,,,,,,,,,,,,, +util_joint_auto_tour_mode_ASC_shared_ride_2,joint - auto tour mode ASC -- shared ride 2,@(df.is_joint & df.tour_mode_is_auto),,,joint_auto_ASC_sr2,joint_auto_ASC_sr2,,,,,,,,,,,,,,,,, +util_joint_auto_tour_mode_ASC_shared_ride_3_,joint - auto tour mode ASC -- shared ride 3+,@(df.is_joint & df.tour_mode_is_auto),,,,,joint_auto_ASC_sr3p,joint_auto_ASC_sr3p,,,,,,,,,,,,,,, +util_joint_auto_tour_mode_ASC_walk,joint - auto tour mode ASC -- walk,@(df.is_joint & df.tour_mode_is_auto),,,,,,,joint_auto_ASC_walk,,,,,,,,,,,,,, +util_joint_auto_tour_mode_ASC_ride_hail,joint - auto tour mode ASC -- ride hail,@(df.is_joint & df.tour_mode_is_ride_hail),,,,,,,,,,,,,,,,,,,joint_auto_ASC_rh,joint_auto_ASC_rh,joint_auto_ASC_rh +util_joint_Walk_tour_mode_ASC_ride_hail,joint - Walk tour mode ASC -- ride hail,@(df.is_joint & df.tour_mode_is_ride_hail),,,,,,,joint_walk_ASC_rh,,,,,,,,,,,,,, util_joint_Bike_tour_mode_ASC_walk,joint - Bike tour mode ASC -- walk,@df.is_joint & (df.i_tour_mode == I_BIKE_MODE),,,,,,,joint_bike_ASC_walk,,,,,,,,,,,,,, util_joint_Bike_tour_mode_ASC_ride_hail,joint - Bike tour mode ASC -- ride hail,@df.is_joint & (df.i_tour_mode == I_BIKE_MODE),,,,,,,,,,,,,,,,,,,joint_bike_ASC_rh,joint_bike_ASC_rh,joint_bike_ASC_rh util_joint_Walk_to_Transit_tour_mode_ASC_light_rail,joint - Walk to Transit tour mode ASC -- light rail,@(df.is_joint & df.tour_mode_is_walk_transit & ~df.walk_ferry_available),,,,,,,,,,joint_walk_transit_ASC_lightrail,,,,,,,,,,, @@ -394,9 +394,9 @@ util_joint_Ride_Hail_tour_mode_ASC_shared_ride_2,joint - Ride Hail tour mode ASC util_joint_Ride_Hail_tour_mode_ASC_shared_ride_3_plus,joint - Ride Hail tour mode ASC -- shared ride 3+,@(df.is_joint & df.tour_mode_is_ride_hail),,,,,joint_ride_hail_ASC_sr3p,joint_ride_hail_ASC_sr3p,,,,,,,,,,,,,,, util_joint_Ride_Hail_tour_mode_ASC_walk,joint - Ride Hail tour mode ASC -- walk,@(df.is_joint & df.tour_mode_is_ride_hail),,,,,,,joint_ride_hail_ASC_walk,,,,,,,,,,,,,, util_joint_Ride_Hail_tour_mode_ASC_walk_to_transit,joint - Ride Hail tour mode ASC -- walk to transit,@(df.is_joint & df.tour_mode_is_ride_hail),,,,,,,,,joint_ride_hail_ASC_walk_transit,joint_ride_hail_ASC_walk_transit,joint_ride_hail_ASC_walk_transit,joint_ride_hail_ASC_walk_transit,joint_ride_hail_ASC_walk_transit,,,,,,,, -util_joint_Ride_Hail_tour_mode_ASC_ride_hail_taxi,joint - Ride Hail tour mode ASC -- ride hail,@(df.is_joint & df.i_tour_mode.isin(I_RIDE_HAIL_MODES)),,,,,,,,,,,,,,,,,,,joint_ride_hail_ASC_taxi,, -util_joint_Ride_Hail_tour_mode_ASC_ride_hail_single,joint - Ride Hail tour mode ASC -- ride hail,@(df.is_joint & df.i_tour_mode.isin(I_RIDE_HAIL_MODES)),,,,,,,,,,,,,,,,,,,,joint_ride_hail_ASC_tnc_single, -util_joint_Ride_Hail_tour_mode_ASC_ride_hail_shared,joint - Ride Hail tour mode ASC -- ride hail,@(df.is_joint & df.i_tour_mode.isin(I_RIDE_HAIL_MODES)),,,,,,,,,,,,,,,,,,,,,joint_ride_hail_ASC_tnc_shared +util_joint_Ride_Hail_tour_mode_ASC_ride_hail_taxi,joint - Ride Hail tour mode ASC -- ride hail,@(df.is_joint & df.tour_mode_is_ride_hail),,,,,,,,,,,,,,,,,,,joint_ride_hail_ASC_taxi,, +util_joint_Ride_Hail_tour_mode_ASC_ride_hail_single,joint - Ride Hail tour mode ASC -- ride hail,@(df.is_joint & df.tour_mode_is_ride_hail),,,,,,,,,,,,,,,,,,,,joint_ride_hail_ASC_tnc_single, +util_joint_Ride_Hail_tour_mode_ASC_ride_hail_shared,joint - Ride Hail tour mode ASC -- ride hail,@(df.is_joint & df.tour_mode_is_ride_hail),,,,,,,,,,,,,,,,,,,,,joint_ride_hail_ASC_tnc_shared #,#,,,,,,,,,,,,,,,,,,,,,, util_Walk_not_available_for_long_distances,Walk not available for long distances,@df.tour_mode_is_walk & (od_skims['DISTWALK'] > 3),,,,,,,-999,,,,,,,,,,,,,, util_Bike_not_available_for_long_distances,Bike not available for long distances,@df.tour_mode_is_walk & (od_skims['DISTBIKE'] > 8),,,,,,,,-999,,,,,,,,,,,,, diff --git a/activitysim/examples/prototype_mtc/configs/trip_mode_choice_annotate_trips_preprocessor.csv b/activitysim/examples/prototype_mtc/configs/trip_mode_choice_annotate_trips_preprocessor.csv index b0b82a22b4..edab18258d 100644 --- a/activitysim/examples/prototype_mtc/configs/trip_mode_choice_annotate_trips_preprocessor.csv +++ b/activitysim/examples/prototype_mtc/configs/trip_mode_choice_annotate_trips_preprocessor.csv @@ -12,6 +12,8 @@ Description,Target,Expression #,, ,i_tour_mode,df.tour_mode.map(I_MODE_MAP) ,tour_mode_is_SOV,i_tour_mode.isin(I_SOV_MODES) +,tour_mode_is_SR2,i_tour_mode.isin(I_SR2_MODES) +,tour_mode_is_SR3P,i_tour_mode.isin(I_SR3P_MODES) ,tour_mode_is_auto,i_tour_mode.isin(I_AUTO_MODES) ,tour_mode_is_walk,i_tour_mode == I_WALK_MODE ,tour_mode_is_bike,i_tour_mode == I_BIKE_MODE diff --git a/activitysim/examples/prototype_mtc/configs_chunktrain/logging.yaml b/activitysim/examples/prototype_mtc/configs_chunktrain/logging.yaml new file mode 100644 index 0000000000..74fc31defa --- /dev/null +++ b/activitysim/examples/prototype_mtc/configs_chunktrain/logging.yaml @@ -0,0 +1,76 @@ +# Config for logging +# ------------------ +# See http://docs.python.org/2.7/library/logging.config.html#configuration-dictionary-schema + +logging: + version: 1 + disable_existing_loggers: true + + + # Configuring the default (root) logger is highly recommended + root: + level: DEBUG + handlers: [console, logfile] + + loggers: + + activitysim: + level: DEBUG + handlers: [console, logfile] + propagate: false + + orca: + level: WARNING + handlers: [console, logfile] + propagate: false + + filelock: + level: WARN + handlers: [console, logfile] + propagate: false + + sharrow: + level: INFO + handlers: [console, logfile] + propagate: false + + black: + level: WARN + + blib2to3: + level: WARN + + handlers: + + logfile: + class: logging.FileHandler + filename: !!python/object/apply:activitysim.core.config.log_file_path ['activitysim.log'] + mode: w + formatter: fileFormatter + level: NOTSET + + console: + class: logging.StreamHandler + stream: ext://sys.stdout + formatter: elapsedFormatter + level: !!python/object/apply:activitysim.core.mp_tasks.if_sub_task [WARNING, NOTSET] + + formatters: + + simpleFormatter: + class: logging.Formatter + #format: '%(processName)-10s %(levelname)s - %(name)s - %(message)s' + format: !!python/object/apply:activitysim.core.mp_tasks.if_sub_task [ + '%(processName)-10s %(levelname)s - %(name)s - %(message)s', + '%(levelname)s - %(name)s - %(message)s'] + datefmt: '%d/%m/%Y %H:%M:%S' + + fileFormatter: + class: logging.Formatter + format: '%(asctime)s - %(levelname)s - %(name)s - %(message)s' + datefmt: '%d/%m/%Y %H:%M:%S' + + elapsedFormatter: + (): activitysim.core.tracing.ElapsedTimeFormatter + format: '[{elapsedTime}] {levelname:s}: {message:s}' + style: '{' diff --git a/activitysim/examples/prototype_mtc/configs_chunktrain/settings.yaml b/activitysim/examples/prototype_mtc/configs_chunktrain/settings.yaml new file mode 100644 index 0000000000..ecd18ec9ea --- /dev/null +++ b/activitysim/examples/prototype_mtc/configs_chunktrain/settings.yaml @@ -0,0 +1,85 @@ + +inherit_settings: True + +# raise error if any sub-process fails without waiting for others to complete +fail_fast: True + +chunk_training_mode: training +multiprocess: True +strict: False +use_shadow_pricing: False + +households_sample_size: 4000 +chunk_size: 40_000_000_000 +num_processes: 8 + +# - ------------------------- + +# not recommended or supported for multiprocessing +want_dest_choice_sample_tables: False + + +# - tracing +# trace_hh_id: +# trace_od: + +# to resume after last successful checkpoint, specify resume_after: _ +# resume_after: trip_purpose_and_destination + +models: + ### mp_initialize step + - initialize_landuse + - initialize_households + ### mp_accessibility step + - compute_accessibility + ### mp_households step + - school_location + - workplace_location + - auto_ownership_simulate + - free_parking + - cdap_simulate + - mandatory_tour_frequency + - mandatory_tour_scheduling + - joint_tour_frequency + - joint_tour_composition + - joint_tour_participation + - joint_tour_destination + - joint_tour_scheduling + - non_mandatory_tour_frequency + - non_mandatory_tour_destination + - non_mandatory_tour_scheduling + - tour_mode_choice_simulate + - atwork_subtour_frequency + - atwork_subtour_destination + - atwork_subtour_scheduling + - atwork_subtour_mode_choice + - stop_frequency + - trip_purpose + - trip_destination + - trip_purpose_and_destination + - trip_scheduling + - trip_mode_choice + ### mp_summarize step + - write_data_dictionary + - write_trip_matrices + - write_tables + +multiprocess_steps: + - name: mp_initialize + begin: initialize_landuse + - name: mp_accessibility + begin: compute_accessibility + slice: + tables: + - accessibility + # don't slice any tables not explicitly listed above in slice.tables + except: True + - name: mp_households + begin: school_location + slice: + tables: + - households + - persons + - name: mp_summarize + begin: write_data_dictionary + diff --git a/activitysim/examples/prototype_mtc/configs_mp/logging.yaml b/activitysim/examples/prototype_mtc/configs_mp/logging.yaml index e932009c5d..13e533abba 100644 --- a/activitysim/examples/prototype_mtc/configs_mp/logging.yaml +++ b/activitysim/examples/prototype_mtc/configs_mp/logging.yaml @@ -9,21 +9,33 @@ logging: # Configuring the default (root) logger is highly recommended root: - level: DEBUG + level: NOTSET handlers: [console, logfile] loggers: activitysim: - level: DEBUG + level: INFO handlers: [console, logfile] propagate: false orca: - level: WARNING + level: WARN handlers: [console, logfile] propagate: false + filelock: + level: WARN + + sharrow: + level: INFO + + blib2to3: + level: WARN + + black: + level: WARN + handlers: logfile: @@ -36,18 +48,14 @@ logging: console: class: logging.StreamHandler stream: ext://sys.stdout - formatter: simpleFormatter - #level: NOTSET level: !!python/object/apply:activitysim.core.mp_tasks.if_sub_task [WARNING, NOTSET] + formatter: elapsedFormatter formatters: simpleFormatter: class: logging.Formatter - #format: '%(processName)-10s %(levelname)s - %(name)s - %(message)s' - format: !!python/object/apply:activitysim.core.mp_tasks.if_sub_task [ - '%(processName)-10s %(levelname)s - %(name)s - %(message)s', - '%(levelname)s - %(name)s - %(message)s'] + format: '%(levelname)s - %(message)s' datefmt: '%d/%m/%Y %H:%M:%S' fileFormatter: @@ -55,3 +63,7 @@ logging: format: '%(asctime)s - %(levelname)s - %(name)s - %(message)s' datefmt: '%d/%m/%Y %H:%M:%S' + elapsedFormatter: + (): activitysim.core.tracing.ElapsedTimeFormatter + format: '[{elapsedTime}] {levelname:s}: {message:s}' + style: '{' diff --git a/activitysim/examples/prototype_mtc/configs_production/logging.yaml b/activitysim/examples/prototype_mtc/configs_production/logging.yaml new file mode 100644 index 0000000000..8939607c8f --- /dev/null +++ b/activitysim/examples/prototype_mtc/configs_production/logging.yaml @@ -0,0 +1,71 @@ +# Config for logging +# ------------------ +# See http://docs.python.org/2.7/library/logging.config.html#configuration-dictionary-schema + +logging: + version: 1 + disable_existing_loggers: true + + + # Configuring the default (root) logger is highly recommended + root: + level: DEBUG + handlers: [console, logfile] + + loggers: + + activitysim: + level: DEBUG + handlers: [console, logfile] + propagate: false + + orca: + level: WARNING + handlers: [console, logfile] + propagate: false + + filelock: + level: WARN + handlers: [console, logfile] + propagate: false + + sharrow: + level: INFO + handlers: [console, logfile] + propagate: false + + handlers: + + logfile: + class: logging.FileHandler + filename: !!python/object/apply:activitysim.core.config.log_file_path ['activitysim.log'] + mode: w + formatter: fileFormatter + level: NOTSET + + console: + class: logging.StreamHandler + stream: ext://sys.stdout + formatter: elapsedFormatter + level: !!python/object/apply:activitysim.core.mp_tasks.if_sub_task [WARNING, NOTSET] + + formatters: + + simpleFormatter: + class: logging.Formatter + #format: '%(processName)-10s %(levelname)s - %(name)s - %(message)s' + format: !!python/object/apply:activitysim.core.mp_tasks.if_sub_task [ + '%(processName)-10s %(levelname)s - %(name)s - %(message)s', + '%(levelname)s - %(name)s - %(message)s'] + datefmt: '%d/%m/%Y %H:%M:%S' + + fileFormatter: + class: logging.Formatter + format: '%(asctime)s - %(levelname)s - %(name)s - %(message)s' + datefmt: '%d/%m/%Y %H:%M:%S' + + elapsedFormatter: + (): activitysim.core.tracing.ElapsedTimeFormatter + format: '[{elapsedTime}] {levelname:s}: {message:s}' + style: '{' + diff --git a/activitysim/examples/prototype_mtc/configs_production/settings.yaml b/activitysim/examples/prototype_mtc/configs_production/settings.yaml new file mode 100644 index 0000000000..8b3f59cb57 --- /dev/null +++ b/activitysim/examples/prototype_mtc/configs_production/settings.yaml @@ -0,0 +1,85 @@ + +inherit_settings: True + +# raise error if any sub-process fails without waiting for others to complete +fail_fast: True + +chunk_training_mode: production +multiprocess: True +strict: False +use_shadow_pricing: False + +households_sample_size: 0 +chunk_size: 40_000_000_000 +num_processes: 8 + +# - ------------------------- + +# not recommended or supported for multiprocessing +want_dest_choice_sample_tables: False + + +# - tracing +# trace_hh_id: +# trace_od: + +# to resume after last successful checkpoint, specify resume_after: _ +# resume_after: trip_purpose_and_destination + +models: + ### mp_initialize step + - initialize_landuse + - initialize_households + ### mp_accessibility step + - compute_accessibility + ### mp_households step + - school_location + - workplace_location + - auto_ownership_simulate + - free_parking + - cdap_simulate + - mandatory_tour_frequency + - mandatory_tour_scheduling + - joint_tour_frequency + - joint_tour_composition + - joint_tour_participation + - joint_tour_destination + - joint_tour_scheduling + - non_mandatory_tour_frequency + - non_mandatory_tour_destination + - non_mandatory_tour_scheduling + - tour_mode_choice_simulate + - atwork_subtour_frequency + - atwork_subtour_destination + - atwork_subtour_scheduling + - atwork_subtour_mode_choice + - stop_frequency + - trip_purpose + - trip_destination + - trip_purpose_and_destination + - trip_scheduling + - trip_mode_choice + ### mp_summarize step + - write_data_dictionary + - write_trip_matrices + - write_tables + +multiprocess_steps: + - name: mp_initialize + begin: initialize_landuse + - name: mp_accessibility + begin: compute_accessibility + slice: + tables: + - accessibility + # don't slice any tables not explicitly listed above in slice.tables + except: True + - name: mp_households + begin: school_location + slice: + tables: + - households + - persons + - name: mp_summarize + begin: write_data_dictionary + diff --git a/activitysim/examples/prototype_mtc/configs_sh/settings.yaml b/activitysim/examples/prototype_mtc/configs_sh/settings.yaml new file mode 100644 index 0000000000..9a5a2162c7 --- /dev/null +++ b/activitysim/examples/prototype_mtc/configs_sh/settings.yaml @@ -0,0 +1,3 @@ + +inherit_settings: True +sharrow: require diff --git a/activitysim/examples/prototype_mtc/configs_sh_compile/settings.yaml b/activitysim/examples/prototype_mtc/configs_sh_compile/settings.yaml new file mode 100644 index 0000000000..598f847999 --- /dev/null +++ b/activitysim/examples/prototype_mtc/configs_sh_compile/settings.yaml @@ -0,0 +1,5 @@ + +inherit_settings: True +sharrow: test +chunk_training_mode: disabled +households_sample_size: 100 diff --git a/activitysim/examples/prototype_mtc/test/configs/settings.yaml b/activitysim/examples/prototype_mtc/test/configs/settings.yaml index abaebfa7b6..7626237cad 100644 --- a/activitysim/examples/prototype_mtc/test/configs/settings.yaml +++ b/activitysim/examples/prototype_mtc/test/configs/settings.yaml @@ -23,3 +23,5 @@ output_tables: sort: True tables: - trips + +recode_pipeline_columns: False diff --git a/activitysim/examples/prototype_mtc/test/configs_chunkless/settings.yaml b/activitysim/examples/prototype_mtc/test/configs_chunkless/settings.yaml index 3e670aca94..7639c323e4 100644 --- a/activitysim/examples/prototype_mtc/test/configs_chunkless/settings.yaml +++ b/activitysim/examples/prototype_mtc/test/configs_chunkless/settings.yaml @@ -24,3 +24,5 @@ output_tables: sort: True tables: - trips + +recode_pipeline_columns: False diff --git a/activitysim/examples/prototype_mtc/test/configs_mp/settings.yaml b/activitysim/examples/prototype_mtc/test/configs_mp/settings.yaml index 88c0311935..038c09a5f6 100644 --- a/activitysim/examples/prototype_mtc/test/configs_mp/settings.yaml +++ b/activitysim/examples/prototype_mtc/test/configs_mp/settings.yaml @@ -29,3 +29,5 @@ output_tables: sort: True tables: - trips + +recode_pipeline_columns: False diff --git a/activitysim/examples/prototype_mtc/test/configs_recode/network_los.yaml b/activitysim/examples/prototype_mtc/test/configs_recode/network_los.yaml new file mode 100644 index 0000000000..1c4cd79daf --- /dev/null +++ b/activitysim/examples/prototype_mtc/test/configs_recode/network_los.yaml @@ -0,0 +1,6 @@ +inherit_settings: True + +# read cached skims (using numpy memmap) from output directory (memmap is faster than omx ) +read_skim_cache: False +# write memmapped cached skims to output directory after reading from omx, for use in subsequent runs +write_skim_cache: False diff --git a/activitysim/examples/prototype_mtc/test/configs_recode/settings.yaml b/activitysim/examples/prototype_mtc/test/configs_recode/settings.yaml new file mode 100644 index 0000000000..c563f13123 --- /dev/null +++ b/activitysim/examples/prototype_mtc/test/configs_recode/settings.yaml @@ -0,0 +1,30 @@ +inherit_settings: True + +# treat warnings as errors +strict: True + +# number of households to simulate +households_sample_size: 10 +chunk_size: 0 + +# - shadow pricing global switches +use_shadow_pricing: False + +# turn writing of sample_tables on and off for all models +# (if True, tables will be written if DEST_CHOICE_SAMPLE_TABLE_NAME is specified in individual model settings) +want_dest_choice_sample_tables: False + +cleanup_pipeline_after_run: True + +output_tables: + h5_store: False + action: include + prefix: final_ + sort: True + tables: + - tablename: trips + decode_columns: + origin: land_use.zone_id + destination: land_use.zone_id + +recode_pipeline_columns: True diff --git a/activitysim/examples/prototype_mtc/test/configs_sharrow/network_los.yaml b/activitysim/examples/prototype_mtc/test/configs_sharrow/network_los.yaml new file mode 100644 index 0000000000..1c4cd79daf --- /dev/null +++ b/activitysim/examples/prototype_mtc/test/configs_sharrow/network_los.yaml @@ -0,0 +1,6 @@ +inherit_settings: True + +# read cached skims (using numpy memmap) from output directory (memmap is faster than omx ) +read_skim_cache: False +# write memmapped cached skims to output directory after reading from omx, for use in subsequent runs +write_skim_cache: False diff --git a/activitysim/examples/prototype_mtc/test/configs_sharrow/settings.yaml b/activitysim/examples/prototype_mtc/test/configs_sharrow/settings.yaml new file mode 100644 index 0000000000..3bfe65d0f8 --- /dev/null +++ b/activitysim/examples/prototype_mtc/test/configs_sharrow/settings.yaml @@ -0,0 +1,31 @@ +inherit_settings: True + +# treat warnings as errors +strict: True + +# number of households to simulate +households_sample_size: 10 +chunk_size: 0 + +# - shadow pricing global switches +use_shadow_pricing: False + +# turn writing of sample_tables on and off for all models +# (if True, tables will be written if DEST_CHOICE_SAMPLE_TABLE_NAME is specified in individual model settings) +want_dest_choice_sample_tables: False + +cleanup_pipeline_after_run: True + +output_tables: + h5_store: False + action: include + prefix: final_ + sort: True + tables: + - tablename: trips + decode_columns: + origin: land_use.zone_id + destination: land_use.zone_id + +sharrow: require +recode_pipeline_columns: True diff --git a/activitysim/examples/prototype_mtc/test/test_mtc.py b/activitysim/examples/prototype_mtc/test/test_mtc.py index fb5751a297..b79266f8fd 100644 --- a/activitysim/examples/prototype_mtc/test/test_mtc.py +++ b/activitysim/examples/prototype_mtc/test/test_mtc.py @@ -2,6 +2,7 @@ # See full license in LICENSE.txt. import os import subprocess +import sys import pandas as pd import pandas.testing as pdt @@ -15,7 +16,7 @@ def teardown_function(func): inject.reinject_decorated_tables() -def run_test_mtc(multiprocess=False, chunkless=False): +def run_test_mtc(multiprocess=False, chunkless=False, recode=False, sharrow=False): def example_path(dirname): resource = os.path.join("examples", "prototype_mtc", dirname) return pkg_resources.resource_filename("activitysim", resource) @@ -58,6 +59,28 @@ def regress(): "-o", test_path("output"), ] + elif recode: + run_args = [ + "-c", + test_path("configs_recode"), + "-c", + example_path("configs"), + "-d", + example_path("data"), + "-o", + test_path("output"), + ] + elif sharrow: + run_args = [ + "-c", + test_path("configs_sharrow"), + "-c", + example_path("configs"), + "-d", + example_path("data"), + "-o", + test_path("output"), + ] else: run_args = [ "-c", @@ -70,7 +93,10 @@ def regress(): test_path("output"), ] - subprocess.run(["coverage", "run", "-a", file_path] + run_args, check=True) + if os.environ.get("GITHUB_ACTIONS") == "true": + subprocess.run(["coverage", "run", "-a", file_path] + run_args, check=True) + else: + subprocess.run([sys.executable, file_path] + run_args, check=True) regress() @@ -87,8 +113,18 @@ def test_mtc_mp(): run_test_mtc(multiprocess=True) +def test_mtc_recode(): + run_test_mtc(recode=True) + + +def test_mtc_sharrow(): + run_test_mtc(sharrow=True) + + if __name__ == "__main__": run_test_mtc(multiprocess=False) run_test_mtc(multiprocess=True) run_test_mtc(multiprocess=False, chunkless=True) + run_test_mtc(recode=True) + run_test_mtc(sharrow=True) diff --git a/activitysim/examples/prototype_mtc_extended/configs/settings.yaml b/activitysim/examples/prototype_mtc_extended/configs/settings.yaml index b1faa5c258..e12237fe75 100644 --- a/activitysim/examples/prototype_mtc_extended/configs/settings.yaml +++ b/activitysim/examples/prototype_mtc_extended/configs/settings.yaml @@ -26,6 +26,8 @@ input_table_list: workers: num_workers VEHICL: auto_ownership TAZ: home_zone_id + recode_columns: + home_zone_id: land_use.zone_id keep_columns: - home_zone_id - income @@ -58,6 +60,8 @@ input_table_list: rename_columns: TAZ: zone_id # person_id is the required index column COUNTY: county_id + recode_columns: + zone_id: zero-based keep_columns: - DISTRICT - SD @@ -222,11 +226,25 @@ output_tables: tables: - checkpoints - accessibility - - land_use - - households - - persons - - tours - - trips + - tablename: land_use + decode_columns: + zone_id: land_use.zone_id + - tablename: households + decode_columns: + home_zone_id: land_use.zone_id + - tablename: persons + decode_columns: + home_zone_id: land_use.zone_id + school_zone_id: nonnegative | land_use.zone_id + workplace_zone_id: nonnegative | land_use.zone_id + - tablename: tours + decode_columns: + origin: land_use.zone_id + destination: land_use.zone_id + - tablename: trips + decode_columns: + origin: land_use.zone_id + destination: land_use.zone_id - joint_tour_participants - vehicles diff --git a/activitysim/examples/prototype_mtc_extended/configs_sharrow/settings.yaml b/activitysim/examples/prototype_mtc_extended/configs_sharrow/settings.yaml new file mode 100644 index 0000000000..7ddeaad9cf --- /dev/null +++ b/activitysim/examples/prototype_mtc_extended/configs_sharrow/settings.yaml @@ -0,0 +1,3 @@ +inherit_settings: True +sharrow: require +recode_pipeline_columns: True diff --git a/activitysim/examples/prototype_mtc_extended/test/configs/settings.yaml b/activitysim/examples/prototype_mtc_extended/test/configs/settings.yaml index 78bdcf9220..2b56a6ec8d 100644 --- a/activitysim/examples/prototype_mtc_extended/test/configs/settings.yaml +++ b/activitysim/examples/prototype_mtc_extended/test/configs/settings.yaml @@ -36,5 +36,8 @@ output_tables: prefix: final_ sort: True tables: - - trips + - tablename: trips + decode_columns: + origin: land_use.zone_id + destination: land_use.zone_id - vehicles diff --git a/activitysim/examples/prototype_mtc_extended/test/test_mtc_extended.py b/activitysim/examples/prototype_mtc_extended/test/test_mtc_extended.py index cc7bb7b546..1f0e0e0168 100644 --- a/activitysim/examples/prototype_mtc_extended/test/test_mtc_extended.py +++ b/activitysim/examples/prototype_mtc_extended/test/test_mtc_extended.py @@ -2,6 +2,7 @@ # See full license in LICENSE.txt. import os import subprocess +import sys import pandas as pd import pandas.testing as pdt @@ -15,7 +16,7 @@ def teardown_function(func): inject.reinject_decorated_tables() -def test_prototype_mtc_extended(): +def _test_prototype_mtc_extended(sharrow=False): def example_path(dirname): resource = os.path.join("examples", "prototype_mtc_extended", dirname) return pkg_resources.resource_filename("activitysim", resource) @@ -41,30 +42,40 @@ def regress(): pdt.assert_frame_equal(final_vehicles_df, regress_vehicles_df) file_path = os.path.join(os.path.dirname(__file__), "simulation.py") - - subprocess.run( - [ - "coverage", - "run", - "-a", - file_path, - "-c", - test_path("configs"), - "-c", - example_path("configs"), - "-c", - example_mtc_path("configs"), - "-d", - example_mtc_path("data"), - "-o", - test_path("output"), - ], - check=True, - ) + if sharrow: + sh_configs = ["-c", example_path("configs_sharrow")] + else: + sh_configs = [] + run_args = sh_configs + [ + "-c", + test_path("configs"), + "-c", + example_path("configs"), + "-c", + example_mtc_path("configs"), + "-d", + example_mtc_path("data"), + "-o", + test_path("output"), + ] + if os.environ.get("GITHUB_ACTIONS") == "true": + subprocess.run(["coverage", "run", "-a", file_path] + run_args, check=True) + else: + subprocess.run( + [sys.executable, "-m", "activitysim", "run"] + run_args, check=True + ) regress() +def test_prototype_mtc_extended(): + _test_prototype_mtc_extended(sharrow=False) + + +def test_prototype_mtc_extended_sharrow(): + _test_prototype_mtc_extended(sharrow=True) + + if __name__ == "__main__": test_prototype_mtc_extended() diff --git a/activitysim/examples/prototype_mwcog/configs/legacy-1.0.4/settings.yaml b/activitysim/examples/prototype_mwcog/configs/legacy-1.0.4/settings.yaml new file mode 100644 index 0000000000..4f81102371 --- /dev/null +++ b/activitysim/examples/prototype_mwcog/configs/legacy-1.0.4/settings.yaml @@ -0,0 +1,155 @@ +# input tables +# +# activitysim uses "well-known" index and foreign key names for imported tables (e.g. households, persons, land_use) +# as well as for created tables (tours, joint_tour_participants, trips) +# e.g. the households table must have an index column 'household_id' and the foreign key to households in the +# persons table is also household_id. This naming convention allows activitysim to intuit the relationship +# between tables - for instance, to ensure that multiprocess slicing includes all the persons, tours, and trips +# in the same subprocess pipeline. The same strategy is also when chunking choosers, and to support tracing by +# household_id. +# +# the input_table_list index_col directive instructs activitysim to set the imported table index to zone_id +# you cannot change the well-known name of the index by modifying this directive. However, if your input file +# has a different id column name, you can rename it to the required index name with the rename_columns directive. +# In the settings below, the 'TAZ' column in the imported table is renamed 'zone_id' in the rename_columns settings. +# +# input tables +input_table_list: + # + # households (table index 'household_id') + # + - tablename: households + filename: combined_synthetic_hh_2018.csv + index_col: household_id + rename_columns: + NP: hhsize + VEH: auto_ownership + TAZ: home_zone_id + # + # persons (table index 'person_id') + # + - tablename: persons + filename: combined_synthetic_per_2018.csv + index_col: person_id + rename_columns: + AGEP: age + hh_id: household_id + # + # land_use (table index 'zone_id') + # + - tablename: land_use + filename: LU_taz3722_rnd91a_2018_adj_enrollment.csv + index_col: zone_id + rename_columns: + TAZ: zone_id + +# convert input CSVs to HDF5 format and save to outputs directory +# create_input_store: True + +# number of households to simulate +households_sample_size: 100 +# simulate all households +#households_sample_size: 0 + +#household_ids: override_household_ids.csv + +chunk_size: 0 + +# set false to disable variability check in simple_simulate and interaction_simulate +check_for_variability: False + +# - shadow pricing global switches + +# turn shadow_pricing on and off for all models (e.g. school and work) +# shadow pricing is deprecated for less than full samples +# see shadow_pricing.yaml for additional settings +use_shadow_pricing: False + + +# - tracing + +# trace household id; comment out or leave empty for no trace +# households with all tour types +# [ 728370 1234067 1402924 1594625 1595333 1747572 1896849 1931818 2222690 2344951 2677154] +#trace_hh_id: 94944 + +# trace origin, destination in accessibility calculation; comment out or leave empty for no trace +# trace_od: [5, 11] +trace_od: + + +models: + - initialize_landuse + - compute_accessibility + - initialize_households + - school_location + - workplace_location + - work_from_home + - transit_pass_subsidy + - transit_pass_ownership + - auto_ownership_simulate + - free_parking + - telecommute_frequency + - cdap_simulate + - mandatory_tour_frequency + - mandatory_tour_scheduling + - joint_tour_frequency + - joint_tour_composition + - joint_tour_participation + - joint_tour_destination + - joint_tour_scheduling + - non_mandatory_tour_frequency + - non_mandatory_tour_destination + - non_mandatory_tour_scheduling + - tour_mode_choice_simulate + - atwork_subtour_frequency + - atwork_subtour_destination + - atwork_subtour_scheduling + - atwork_subtour_mode_choice + - stop_frequency + - trip_purpose + - trip_destination + - trip_purpose_and_destination + - trip_scheduling + - trip_mode_choice + - write_data_dictionary + - track_skim_usage + - write_tables + - write_trip_matrices + +# to resume after last successful checkpoint, specify resume_after: _ +resume_after: + +output_tables: + h5_store: False + action: include + prefix: final_ + tables: + - checkpoints + - accessibility + - land_use + - households + - persons + - tours + - trips + - joint_tour_participants + +# area_types less than this are considered urban +urban_threshold: 4 +cbd_threshold: 2 +rural_threshold: 6 + +# - value of time + +# value_of_time = lognormal(np.log(median_value_of_time * mu), sigma).clip(min_vot, max_vot) + +min_value_of_time: 1 +max_value_of_time: 50 +distributed_vot_mu: 0.684 +distributed_vot_sigma: 0.85 + +household_median_value_of_time: + 1: 6.01 + 2: 8.81 + 3: 10.44 + 4: 12.86 diff --git a/activitysim/examples/prototype_mwcog/configs/logging.yaml b/activitysim/examples/prototype_mwcog/configs/logging.yaml index 6dfca578a4..c78f442387 100644 --- a/activitysim/examples/prototype_mwcog/configs/logging.yaml +++ b/activitysim/examples/prototype_mwcog/configs/logging.yaml @@ -15,7 +15,7 @@ logging: loggers: activitysim: - level: DEBUG + level: INFO handlers: [console, logfile] propagate: false @@ -24,6 +24,18 @@ logging: handlers: [console, logfile] propagate: false + filelock: + level: WARN + + sharrow: + level: INFO + + blib2to3: + level: WARN + + black: + level: WARN + handlers: logfile: @@ -36,7 +48,7 @@ logging: console: class: logging.StreamHandler stream: ext://sys.stdout - formatter: simpleFormatter + formatter: elapsedFormatter level: NOTSET formatters: @@ -51,3 +63,8 @@ logging: class: logging.Formatter format: '%(asctime)s - %(levelname)s - %(name)s - %(message)s' datefmt: '%d/%m/%Y %H:%M:%S' + + elapsedFormatter: + (): activitysim.core.tracing.ElapsedTimeFormatter + format: '[{elapsedTime}] {levelname:s}: {message:s}' + style: '{' diff --git a/activitysim/examples/prototype_mwcog/configs/mandatory_tour_scheduling.yaml b/activitysim/examples/prototype_mwcog/configs/mandatory_tour_scheduling.yaml index 120959d36f..ceda1517dc 100644 --- a/activitysim/examples/prototype_mwcog/configs/mandatory_tour_scheduling.yaml +++ b/activitysim/examples/prototype_mwcog/configs/mandatory_tour_scheduling.yaml @@ -20,7 +20,7 @@ SIMULATE_CHOOSER_COLUMNS: - school_zone_id - home_zone_id - TAZ - + LOGSUM_SETTINGS: tour_mode_choice.yaml TOUR_SPEC_SEGMENTS: @@ -28,6 +28,10 @@ TOUR_SPEC_SEGMENTS: school: school univ: univ +preprocessor: + SPEC: mandatory_tour_scheduling_annotate_choosers_preprocessor.csv + DF: df + ALTS_PREPROCESSOR: work: SPEC: mandatory_tour_scheduling_annotate_tours_preprocessor.csv diff --git a/activitysim/examples/prototype_mwcog/configs/mandatory_tour_scheduling_annotate_choosers_preprocessor.csv b/activitysim/examples/prototype_mwcog/configs/mandatory_tour_scheduling_annotate_choosers_preprocessor.csv new file mode 100644 index 0000000000..9b6fde9c9f --- /dev/null +++ b/activitysim/examples/prototype_mwcog/configs/mandatory_tour_scheduling_annotate_choosers_preprocessor.csv @@ -0,0 +1,3 @@ +Description,Target,Expression +subsequent_tour_is_work,subsequent_tour_is_work,df.tour_type.shift(-1) == 'work' +subsequent_tour_is_school,subsequent_tour_is_school,df.tour_type.shift(-1) == 'school' diff --git a/activitysim/examples/prototype_mwcog/configs/non_mandatory_tour_frequency.csv b/activitysim/examples/prototype_mwcog/configs/non_mandatory_tour_frequency.csv index 6abd71a70b..cb99f18402 100644 --- a/activitysim/examples/prototype_mwcog/configs/non_mandatory_tour_frequency.csv +++ b/activitysim/examples/prototype_mwcog/configs/non_mandatory_tour_frequency.csv @@ -215,9 +215,9 @@ util_calibration_mand_gt0_nonmandtours_gt3,Mandatory tours > 0 and Non-mandatory util_calibration_mand_0_nonmandtours_2,Mandatory tours = 0 and Non-mandatory tours = 2 Calibration Constant,(num_mand == 0)&(tot_tours == 2),coef_calib_nonmandtours_2_mand_0,coef_calib_nonmandtours_2_mand_0,coef_calib_nonmandtours_2_mand_0,coef_calib_nonmandtours_2_mand_0,coef_calib_nonmandtours_2_mand_0,coef_calib_nonmandtours_2_mand_0,coef_calib_nonmandtours_2_mand_0,coef_calib_nonmandtours_2_mand_0 util_calibration_mand_0_nonmandtours_3,Mandatory tours = 0 and Non-mandatory tours = 3 Calibration Constant,(num_mand == 0)&(tot_tours == 3),coef_calib_nonmandtours_3_mand_0,coef_calib_nonmandtours_3_mand_0,coef_calib_nonmandtours_3_mand_0,coef_calib_nonmandtours_3_mand_0,coef_calib_nonmandtours_3_mand_0,coef_calib_nonmandtours_3_mand_0,coef_calib_nonmandtours_3_mand_0,coef_calib_nonmandtours_3_mand_0 util_calibration_mand_0_nonmandtours_gt4,Mandatory tours = 0 and Non-mandatory tours = 4+ Calibration Constant,(num_mand == 0)&(tot_tours >= 4),coef_calib_nonmandtours_gt4_mand_0,coef_calib_nonmandtours_gt4_mand_0,coef_calib_nonmandtours_gt4_mand_0,coef_calib_nonmandtours_gt4_mand_0,coef_calib_nonmandtours_gt4_mand_0,coef_calib_nonmandtours_gt4_mand_0,coef_calib_nonmandtours_gt4_mand_0,coef_calib_nonmandtours_gt4_mand_0 -util_tc_1dayperweek_1,Person that telecommutes 1 day per week,telecommute_frequency=='1_day_week' & tot_tours == 1,coef_tc_1dayperweek_1,coef_tc_1dayperweek_1,coef_tc_1dayperweek_1,0,0,0,0 -util_tc_23dayperweek_1,Person that telecommutes 2 to 3 days per week,telecommute_frequency=='2_3_days_week' & tot_tours == 1,coef_tc_23dayperweek_1,coef_tc_23dayperweek_1,coef_tc_23dayperweek_1,0,0,0,0 -util_tc_4pdayperweek_1,Person that telecommutes 2 to 3 days per week,telecommute_frequency=='4_days_week' & tot_tours == 1,coef_tc_4pdayperweek_1,coef_tc_4pdayperweek_1,coef_tc_4pdayperweek_1,0,0,0,0 -util_tc_1dayperweek_2p,Person that telecommutes 1 day per week,telecommute_frequency=='1_day_week' & tot_tours > 1,coef_tc_1dayperweek_2p,coef_tc_1dayperweek_2p,coef_tc_1dayperweek_2p,0,0,0,0 -util_tc_23dayperweek_2p,Person that telecommutes 2 to 3 days per week,telecommute_frequency=='2_3_days_week' & tot_tours > 1,coef_tc_23dayperweek_2p,coef_tc_23dayperweek_2p,coef_tc_23dayperweek_2p,0,0,0,0 -util_tc_4pdayperweek_2p,Person that telecommutes 2 to 3 days per week,telecommute_frequency=='4_days_week' & tot_tours > 1,coef_tc_4pdayperweek_2p,coef_tc_4pdayperweek_2p,coef_tc_4pdayperweek_2p,0,0,0,0 \ No newline at end of file +util_tc_1dayperweek_1,Person that telecommutes 1 day per week,(telecommute_frequency=='1_day_week') & (tot_tours == 1),coef_tc_1dayperweek_1,coef_tc_1dayperweek_1,coef_tc_1dayperweek_1,0,0,0,0 +util_tc_23dayperweek_1,Person that telecommutes 2 to 3 days per week,(telecommute_frequency=='2_3_days_week') & (tot_tours == 1),coef_tc_23dayperweek_1,coef_tc_23dayperweek_1,coef_tc_23dayperweek_1,0,0,0,0 +util_tc_4pdayperweek_1,Person that telecommutes 2 to 3 days per week,(telecommute_frequency=='4_days_week') & (tot_tours == 1),coef_tc_4pdayperweek_1,coef_tc_4pdayperweek_1,coef_tc_4pdayperweek_1,0,0,0,0 +util_tc_1dayperweek_2p,Person that telecommutes 1 day per week,(telecommute_frequency=='1_day_week') & (tot_tours > 1),coef_tc_1dayperweek_2p,coef_tc_1dayperweek_2p,coef_tc_1dayperweek_2p,0,0,0,0 +util_tc_23dayperweek_2p,Person that telecommutes 2 to 3 days per week,(telecommute_frequency=='2_3_days_week') & (tot_tours > 1),coef_tc_23dayperweek_2p,coef_tc_23dayperweek_2p,coef_tc_23dayperweek_2p,0,0,0,0 +util_tc_4pdayperweek_2p,Person that telecommutes 2 to 3 days per week,(telecommute_frequency=='4_days_week') & (tot_tours > 1),coef_tc_4pdayperweek_2p,coef_tc_4pdayperweek_2p,coef_tc_4pdayperweek_2p,0,0,0,0 diff --git a/activitysim/examples/prototype_mwcog/configs/settings.yaml b/activitysim/examples/prototype_mwcog/configs/settings.yaml index 4f81102371..fc126b771a 100644 --- a/activitysim/examples/prototype_mwcog/configs/settings.yaml +++ b/activitysim/examples/prototype_mwcog/configs/settings.yaml @@ -25,6 +25,8 @@ input_table_list: NP: hhsize VEH: auto_ownership TAZ: home_zone_id + recode_columns: + home_zone_id: land_use.zone_id # # persons (table index 'person_id') # @@ -34,6 +36,8 @@ input_table_list: rename_columns: AGEP: age hh_id: household_id + recode_columns: + TAZ: land_use.zone_id # # land_use (table index 'zone_id') # @@ -42,6 +46,8 @@ input_table_list: index_col: zone_id rename_columns: TAZ: zone_id + recode_columns: + zone_id: zero-based # convert input CSVs to HDF5 format and save to outputs directory # create_input_store: True @@ -126,12 +132,30 @@ output_tables: prefix: final_ tables: - checkpoints - - accessibility - - land_use - - households - - persons - - tours - - trips + - tablename: accessibility + decode_columns: + zone_id: land_use.zone_id + - tablename: land_use + decode_columns: + zone_id: land_use.zone_id + - tablename: households + decode_columns: + home_zone_id: land_use.zone_id + TAZ: land_use.zone_id + - tablename: persons + decode_columns: + home_zone_id: land_use.zone_id + TAZ: land_use.zone_id + school_zone_id: nonnegative | land_use.zone_id + workplace_zone_id: nonnegative | land_use.zone_id + - tablename: tours + decode_columns: + origin: land_use.zone_id + destination: land_use.zone_id + - tablename: trips + decode_columns: + origin: land_use.zone_id + destination: land_use.zone_id - joint_tour_participants # area_types less than this are considered urban @@ -153,3 +177,5 @@ household_median_value_of_time: 2: 8.81 3: 10.44 4: 12.86 + +recode_pipeline_columns: False diff --git a/activitysim/examples/prototype_mwcog/configs/tour_scheduling_atwork.csv b/activitysim/examples/prototype_mwcog/configs/tour_scheduling_atwork.csv index 93d8fc4726..12b6814193 100644 --- a/activitysim/examples/prototype_mwcog/configs/tour_scheduling_atwork.csv +++ b/activitysim/examples/prototype_mwcog/configs/tour_scheduling_atwork.csv @@ -19,22 +19,22 @@ util_Subtour_purpose_Eatout_Departure_after_1230_pm_Linear,Subtour purpose: Eat- util_Subtour_purpose_Eatout_Duration_less_than_0p5_hours_depart_and_arrive_in_the_same_period,Subtour purpose: Eat-out - Duration less than 0.5 hours (depart and arrive in the same period),"@np.where(((df.tour_type == 'eat') & (df.duration<1)), (np.where((df.duration <= 1), np.minimum(1 - df.duration, 47), 0) + np.where((df.duration > 1), np.minimum(df.duration - 1, 47), 0)), 0)",coef_Subtour_purpose_Eatout_Duration_less_than_0p5_hours_depart_and_arrive_in_the_same_period util__Departure_constants,# Departure constants,,coef__Departure_constants util_Shift_for_every_30_minutes_before_1030_am_Linear,Shift for every 30 minutes before 10:30 am - Linear,"@np.where((df.start<16), (np.where((df.start< 16), np.minimum(16 - df.start, 9), 0) + np.where((df.start> 21), np.minimum(df.start - 21, 11), 0)), 0)",coef_Shift_for_every_30_minutes_before_1030_am_Linear -util_Before_1100_AM,Before 11:00 AM,@(df.start<17),coef_start_Before_1100_AM -util_1100_AM_1130_AM,11:00 AM - 11:30 AM,@(df.start==17),coef_start_1100_AM_1130_AM -util_1130_AM_1200_PM,11:30 AM - 12:00 PM,@(df.start==18),coef_start_1130_AM_1200_PM -util_1200_AM_1230_PM,12:00 AM - 12:30 PM,@(df.start==19),coef_start_1200_AM_1230_PM -util_1230_PM_0100_PM,12:30 PM - 01:00 PM,@(df.start==20),coef_start_1230_PM_0100_PM -util_After_0100_PM,After 01:00 PM,@(df.start>20),coef_start_After_0100_PM +util_Before_1100_AM_start,Before 11:00 AM,@(df.start<17),coef_start_Before_1100_AM +util_1100_AM_1130_AM_start,11:00 AM - 11:30 AM,@(df.start==17),coef_start_1100_AM_1130_AM +util_1130_AM_1200_PM_start,11:30 AM - 12:00 PM,@(df.start==18),coef_start_1130_AM_1200_PM +util_1200_AM_1230_PM_start,12:00 AM - 12:30 PM,@(df.start==19),coef_start_1200_AM_1230_PM +util_1230_PM_0100_PM_start,12:30 PM - 01:00 PM,@(df.start==20),coef_start_1230_PM_0100_PM +util_After_0100_PM_start,After 01:00 PM,@(df.start>20),coef_start_After_0100_PM util_Shift_for_every_30_minutes_after_130_pm_Square_Root,Shift for every 30 minutes after 1:30 pm - Square Root,"@np.where((df.start>21), ((np.where((df.start < 16), np.minimum(16 - df.start, 9), 0) + np.where((df.start > 21), np.minimum(df.start - 21, 11), 0))** 0.5), 0)",coef_Shift_for_every_30_minutes_after_130_pm_Square_Root util__Arrival_constants,# Arrival constants,,coef__Arrival_constants util_Shift_for_every_30_minutes_before_1130_am_Linear,Shift for every 30 minutes before 11:30 am - Linear,"@np.where((df.end<18), (np.where((df.end < 14), np.minimum(14 - df.end, 9), 0) + np.where((df.end > 24), np.minimum(df.end - 24, 10), 0)), 0)",coef_Shift_for_every_30_minutes_before_1130_am_Linear -util_Before_1200_PM,Before 12:00 PM,@(df.end<19),coef_end_Before_1200_PM -util_1200_AM_1230_PM,12:00 AM - 12:30 PM,@(df.end==19),coef_end_1200_AM_1230_PM -util_1230_PM_0100_PM,12:30 PM - 01:00 PM,@(df.end==20),coef_end_1230_PM_0100_PM -util_0100_PM_0130_PM,01:00 PM - 01:30 PM,@(df.end==21),coef_end_0100_PM_0130_PM -util_0130_PM_0200_PM,01:30 PM - 02:00 PM,@(df.end==22),coef_end_0130_PM_0200_PM -util_0200_PM_0230_PM,02:00 PM - 02:30 PM,@(df.end==23),coef_end_0200_PM_0230_PM -util_After_0230_PM,After 02:30 PM,@(df.end>23),coef_end_After_0230_PM +util_Before_1200_PM_end,Before 12:00 PM,@(df.end<19),coef_end_Before_1200_PM +util_1200_AM_1230_PM_end,12:00 AM - 12:30 PM,@(df.end==19),coef_end_1200_AM_1230_PM +util_1230_PM_0100_PM_end,12:30 PM - 01:00 PM,@(df.end==20),coef_end_1230_PM_0100_PM +util_0100_PM_0130_PM_end,01:00 PM - 01:30 PM,@(df.end==21),coef_end_0100_PM_0130_PM +util_0130_PM_0200_PM_end,01:30 PM - 02:00 PM,@(df.end==22),coef_end_0130_PM_0200_PM +util_0200_PM_0230_PM_end,02:00 PM - 02:30 PM,@(df.end==23),coef_end_0200_PM_0230_PM +util_After_0230_PM_end,After 02:30 PM,@(df.end>23),coef_end_After_0230_PM util_Shift_for_every_30_minutes_after_300_pm_Linear,Shift for every 30 minutes after 3:00 pm - Linear,"@np.where((df.end>24), (np.where((df.end < 14), np.minimum(14 - df.end, 9), 0) + np.where((df.end > 24), np.minimum(df.end - 24, 10), 0)), 0)",coef_Shift_for_every_30_minutes_after_300_pm_Linear util__Duration_constants,# Duration constants,,coef__Duration_constants util_0_hrs,0 hrs,@(df.duration==0),coef_0_hrs diff --git a/activitysim/examples/prototype_mwcog/configs/tour_scheduling_joint.csv b/activitysim/examples/prototype_mwcog/configs/tour_scheduling_joint.csv index a9f0a51860..60cf794bf5 100644 --- a/activitysim/examples/prototype_mwcog/configs/tour_scheduling_joint.csv +++ b/activitysim/examples/prototype_mwcog/configs/tour_scheduling_joint.csv @@ -95,7 +95,7 @@ util_shop_Departure_Constant_10_00_AM_10_30_AM,SHOPPING - Departure Constant: 10 util_shop_Departure_Constant_10_30_AM_11_00_AM,SHOPPING - Departure Constant: 10:30 AM - 11:00 AM,@((df.tour_category == 'joint') & (df.tour_type == 'shopping') & (df.start==16)),coef_shop_Departure_Constant_10_30_AM_11_00_AM util_shop_Departure_Constant_After_11_00_AM,SHOPPING - Departure Constant: After 11:00 AM,@((df.tour_category == 'joint') & (df.tour_type == 'shopping') & (df.start>16)),coef_shop_Departure_Constant_After_11_00_AM util_shop_Departure_Constant_Shift_for_every_30_minutes_after_11_30_am_Linear,SHOPPING - Departure Constant: Shift for every 30 minutes after 11:30 am - Linear,"@np.where(((df.tour_category == 'joint') & (df.tour_type == 'shopping') & (df.start>17)), (np.where((df.start<12), np.minimum(12-df.start,7),0) + np.where((df.start>17), np.minimum(df.start-17,24), 0)), 0)",coef_shop_Departure_Constant_Shift_for_every_30_minutes_after_11_30_am_Linear -util_shop_Departure_Constant_Shift_for_every_30_minutes_after_11_30_am_Squared,SHOPPING - Departure Constant: Shift for every 30 minutes after 11:30 am - Squared,"@np.where(((df.tour_category == 'joint') & (df.tour_type == 'shopping') & (df.start>17)), ((np.where((df.start<12), np.minimum(12-df.start,7), 0) + np.where((df.start>17), np.minimum(df.start-17,24), 0)) ** 2), 0)",coef_shop_Departure_Constant_Shift_for_every_30_minutes_after_11_30_am_Squared +util_shop_Departure_Constant_Shift_for_every_30_minutes_after_11_30_am_Squared,SHOPPING - Departure Constant: Shift for every 30 minutes after 11:30 am - Squared,"@np.where(((df.tour_category == 'joint') & (df.tour_type == 'shopping') & (df.start>17)), ((np.where((df.start<12), np.minimum(12-df.start,7), 0) + np.where((df.start>17), np.minimum(df.start-17,24), 0)).astype(np.float32) ** 2), 0)",coef_shop_Departure_Constant_Shift_for_every_30_minutes_after_11_30_am_Squared util_shop_Arrival_Constant_Shift_for_every_30_minutes_before_12_00_pm_Linear,SHOPPING - Arrival Constant: Shift for every 30 minutes before 12:00 pm - Linear,"@np.where(((df.tour_category == 'joint') & (df.tour_type == 'shopping') & (df.end<19)), (np.where ((df.end<19), np.minimum(19-df.end,10), 0) + np.where((df.end>38), np.minimum(df.end-38,5), 0)), 0)",coef_shop_Arrival_Constant_Shift_for_every_30_minutes_before_12_00_pm_Linear util_shop_Arrival_Constant_Before_12_30_PM,SHOPPING - Arrival Constant: Before 12:30 PM,@((df.tour_category == 'joint') & (df.tour_type == 'shopping') & (df.end<20)),coef_shop_Arrival_Constant_Before_12_30_PM util_shop_Arrival_Constant_12_30_PM_03_00_PM,SHOPPING - Arrival Constant: 12:30 PM - 03:00 PM,@((df.tour_category == 'joint') & (df.tour_type == 'shopping') & ( df.end>=20) & (df.end<=24)),coef_shop_Arrival_Constant_12_30_PM_03_00_PM @@ -144,7 +144,7 @@ util_maint_Departure_Constant_10_00_AM_10_30_AM,MAINTENANCE - Departure Constant util_maint_Departure_Constant_10_30_AM_11_00_AM,MAINTENANCE - Departure Constant: 10:30 AM - 11:00 AM,@((df.tour_category == 'joint') & (df.tour_type == 'othmaint') & (df.start==16)),coef_maint_Departure_Constant_10_30_AM_11_00_AM util_maint_Departure_Constant_After_11_00_AM,MAINTENANCE - Departure Constant: After 11:00 AM,@((df.tour_category == 'joint') & (df.tour_type == 'othmaint') & (df.start>16)),coef_maint_Departure_Constant_After_11_00_AM util_maint_Departure_Constant_Shift_for_every_30_minutes_after_11_30_am_Linear,MAINTENANCE - Departure Constant: Shift for every 30 minutes after 11:30 am - Linear,"@np.where(((df.tour_category == 'joint') & (df.tour_type == 'othmaint') & (df.start>17)), np.where((df.start<10), np.minimum(10-df.start,7), 0) + np.where((df.start>17), np.minimum(df.start-17,24), 0), 0)",coef_maint_Departure_Constant_Shift_for_every_30_minutes_after_11_30_am_Linear -util_maint_Departure_Constant_Shift_for_every_30_minutes_after_11_30_am_Squared,MAINTENANCE - Departure Constant: Shift for every 30 minutes after 11:30 am - Squared,"@np.where(((df.tour_category == 'joint') & (df.tour_type == 'othmaint') & (df.start>17)), ((np.where((df.start<10), np.minimum(10-df.start,7), 0) + np.where((df.start>17), np.minimum(df.start-17,24), 0)) ** 2), 0)",coef_maint_Departure_Constant_Shift_for_every_30_minutes_after_11_30_am_Squared +util_maint_Departure_Constant_Shift_for_every_30_minutes_after_11_30_am_Squared,MAINTENANCE - Departure Constant: Shift for every 30 minutes after 11:30 am - Squared,"@np.where(((df.tour_category == 'joint') & (df.tour_type == 'othmaint') & (df.start>17)), ((np.where((df.start<10), np.minimum(10-df.start,7), 0) + np.where((df.start>17), np.minimum(df.start-17,24), 0)).astype(np.float32) ** 2), 0)",coef_maint_Departure_Constant_Shift_for_every_30_minutes_after_11_30_am_Squared util_maint_Arrival_Constant_Shift_for_every_30_minutes_before_10_00_am_Linear,MAINTENANCE - Arrival Constant: Shift for every 30 minutes before 10:00 am - Linear,"@np.where(((df.tour_category == 'joint') & (df.tour_type == 'othmaint') & (df.end<15)), (np.where((df.end<15), np.minimum(15-df.end,9), 0) + np.where((df.end>28), np.minimum(df.end-28,16), 0)), 0)",coef_maint_Arrival_Constant_Shift_for_every_30_minutes_before_10_00_am_Linear util_maint_Arrival_Constant_Before_10_30_AM,MAINTENANCE - Arrival Constant: Before 10:30 AM,@((df.tour_category == 'joint') & (df.tour_type == 'othmaint') & (df.end<16)),coef_maint_Arrival_Constant_Before_10_30_AM util_maint_Arrival_Constant_10_30_AM_11_00_AM,MAINTENANCE - Arrival Constant: 10:30 AM - 11:00 AM,@((df.tour_category == 'joint') & (df.tour_type == 'othmaint') & (df.end==16)),coef_maint_Arrival_Constant_10_30_AM_11_00_AM diff --git a/activitysim/examples/prototype_mwcog/configs/tour_scheduling_nonmandatory_othmaint.csv b/activitysim/examples/prototype_mwcog/configs/tour_scheduling_nonmandatory_othmaint.csv index 04f26223ed..370db4e47d 100644 --- a/activitysim/examples/prototype_mwcog/configs/tour_scheduling_nonmandatory_othmaint.csv +++ b/activitysim/examples/prototype_mwcog/configs/tour_scheduling_nonmandatory_othmaint.csv @@ -31,7 +31,7 @@ util_maintenance_departure_constant_10_am_to_10_30_am,MAINTENANCE - Departure Co util_maintenance_departure_constant_10_30_am_to_11_am,MAINTENANCE - Departure Constant: 10:30 AM - 11:00 AM,@((df.start==16)),coef_maintenance_departure_constant_10_30_am_to_11_am util_maintenance_departure_constant_after_11_am,MAINTENANCE - Departure Constant: After 11:00 AM,@((df.start>16)),coef_maintenance_departure_constant_after_11_am util_maintenance_departure_constant_shift_for_every_30_minutes_after_11_30_am_linear,MAINTENANCE - Departure Constant: Shift for every 30 minutes after 11:30 am - Linear,"@np.where(((df.start>17)), np.where((df.start<10), np.minimum(10-df.start,7), 0) + np.where((df.start>17), np.minimum(df.start-17,24), 0), 0)",coef_maintenance_departure_constant_shift_for_every_30_minutes_after_11_30_am_linear -util_maintenance_departure_constant_shift_for_every_30_minutes_after_11_30_am_squared,MAINTENANCE - Departure Constant: Shift for every 30 minutes after 11:30 am - Squared,"@np.where(((df.start>17)), ((np.where((df.start<10), np.minimum(10-df.start,7), 0) + np.where((df.start>17), np.minimum(df.start-17,24), 0)) ** 2), 0)",coef_maintenance_departure_constant_shift_for_every_30_minutes_after_11_30_am_squared +util_maintenance_departure_constant_shift_for_every_30_minutes_after_11_30_am_squared,MAINTENANCE - Departure Constant: Shift for every 30 minutes after 11:30 am - Squared,"@np.where(((df.start>17)), ((np.where((df.start<10), np.minimum(10-df.start,7), 0) + np.where((df.start>17), np.minimum(df.start-17,24), 0)).astype(np.float32) ** 2), 0)",coef_maintenance_departure_constant_shift_for_every_30_minutes_after_11_30_am_squared util_maintenance_arrival_constant_shift_for_every_30_minutes_before_10_am_linear,MAINTENANCE - Arrival Constant: Shift for every 30 minutes before 10:00 am - Linear,"@np.where(((df.end<15)), (np.where((df.end<15), np.minimum(15-df.end,9), 0) + np.where((df.end>28), np.minimum(df.end-28,16), 0)), 0)",coef_maintenance_arrival_constant_shift_for_every_30_minutes_before_10_am_linear util_maintenance_arrival_constant_before_10_30_am,MAINTENANCE - Arrival Constant: Before 10:30 AM,@((df.end<16)),coef_maintenance_arrival_constant_before_10_30_am util_maintenance_arrival_constant_10_30_am_to_11_am,MAINTENANCE - Arrival Constant: 10:30 AM - 11:00 AM,@((df.end==16)),coef_maintenance_arrival_constant_10_30_am_to_11_am diff --git a/activitysim/examples/prototype_mwcog/configs/tour_scheduling_nonmandatory_shopping.csv b/activitysim/examples/prototype_mwcog/configs/tour_scheduling_nonmandatory_shopping.csv index 11211303e6..437314f8d5 100644 --- a/activitysim/examples/prototype_mwcog/configs/tour_scheduling_nonmandatory_shopping.csv +++ b/activitysim/examples/prototype_mwcog/configs/tour_scheduling_nonmandatory_shopping.csv @@ -28,7 +28,7 @@ util_shopping_departure_constant_10_am_to_10_30_am,SHOPPING - Departure Constant util_shopping_departure_constant_10_30_am_to_11_00_am,SHOPPING - Departure Constant: 10:30 AM - 11:00 AM,@((df.start==16)),coef_shopping_departure_constant_10_30_am_to_11_00_am util_shopping_departure_constant_after_11_am,SHOPPING - Departure Constant: After 11:00 AM,@((df.start>16)),coef_shopping_departure_constant_after_11_am util_shopping_departure_constant_shift_for_every_30_minutes_after_11_30_am_linear,SHOPPING - Departure Constant: Shift for every 30 minutes after 11:30 am - Linear,"@np.where(((df.start>17)), (np.where((df.start<12), np.minimum(12-df.start,7),0) + np.where((df.start>17), np.minimum(df.start-17,24), 0)), 0)",coef_shopping_departure_constant_shift_for_every_30_minutes_after_11_30_am_linear -util_shopping_departure_constant_shift_for_every_30_minutes_after_11_30_am_squared,SHOPPING - Departure Constant: Shift for every 30 minutes after 11:30 am - Squared,"@np.where(((df.start>17)), ((np.where((df.start<12), np.minimum(12-df.start,7), 0) + np.where((df.start>17), np.minimum(df.start-17,24), 0)) ** 2), 0)",coef_shopping_departure_constant_shift_for_every_30_minutes_after_11_30_am_squared +util_shopping_departure_constant_shift_for_every_30_minutes_after_11_30_am_squared,SHOPPING - Departure Constant: Shift for every 30 minutes after 11:30 am - Squared,"@np.where(((df.start>17)), ((np.where((df.start<12), np.minimum(12-df.start,7), 0) + np.where((df.start>17), np.minimum(df.start-17,24), 0)).astype(np.float32) ** 2), 0)",coef_shopping_departure_constant_shift_for_every_30_minutes_after_11_30_am_squared util_shopping_arrival_constant_shift_for_every_30_minutes_before_12_pm_linear,SHOPPING - Arrival Constant: Shift for every 30 minutes before 12:00 pm - Linear,"@np.where(((df.end<19)), (np.where ((df.end<19), np.minimum(19-df.end,10), 0) + np.where((df.end>38), np.minimum(df.end-38,5), 0)), 0)",coef_shopping_arrival_constant_shift_for_every_30_minutes_before_12_pm_linear util_shopping_arrival_constant_before_12_30_pm,SHOPPING - Arrival Constant: Before 12:30 PM,@((df.end<20)),coef_shopping_arrival_constant_before_12_30_pm util_shopping_arrival_constant_12_30_pm_to_3_pm,SHOPPING - Arrival Constant: 12:30 PM - 03:00 PM,@(( df.end>=20) & (df.end<=24)),coef_shopping_arrival_constant_12_30_pm_to_3_pm diff --git a/activitysim/examples/prototype_mwcog/configs/tour_scheduling_school.csv b/activitysim/examples/prototype_mwcog/configs/tour_scheduling_school.csv index 4349420401..44b1f75abd 100644 --- a/activitysim/examples/prototype_mwcog/configs/tour_scheduling_school.csv +++ b/activitysim/examples/prototype_mwcog/configs/tour_scheduling_school.csv @@ -22,11 +22,11 @@ util_All_adults_in_the_household_are_fulltime_workers_Departure_before_730_am__L util_All_adults_in_the_household_are_fulltime_workers_Departure_after_800_am_Linear,SCHOOL - All adults in the household are fulltime workers - Departure after 8:00 am - Linear,"@((df.is_all_adults_full_time_workers) & (df.start>10)) *((np.minimum(10-df.start,48)*(df.start<=10)) + (np.minimum(df.start-10,48)*(df.start>10)))",coef_All_adults_in_the_household_are_fulltime_workers_Departure_after_800_am_Linear util_All_adults_in_the_household_are_fulltime_workers_Duration_lt_8hrs,SCHOOL - All adults in the household are fulltime workers - Duration < 8hrs,"@((df.is_all_adults_full_time_workers) & (df.end<27)) * ((np.minimum(27-df.end,48)*(df.end<=27)) + (np.minimum(df.end-27,48)*(df.end>27)))",coef_All_adults_in_the_household_are_fulltime_workers_Duration_lt_8hrs util_All_adults_in_the_household_are_fulltime_workers_Duration_gt_8hrs,SCHOOL - All adults in the household are fulltime workers - Duration > 8hrs,"@((df.is_all_adults_full_time_workers) & (df.end>27)) * ((np.minimum(27-df.end,48)*(df.end<=27)) + (np.minimum(df.end-27,48)*(df.end>27)))",coef_All_adults_in_the_household_are_fulltime_workers_Duration_gt_8hrs -util_Subsequent_tour_is_work_tour_Duration_lt_8_hours,SCHOOL - Subsequent tour is work tour: Duration < 8 hours,"@(((df.tour_count>1) & (df.tour_num == 1) & (df.tour_type.shift(-1) == 'work')) & (df.duration<8)) * ((np.minimum(8-df.duration,47)*(df.duration<=8)) + (np.minimum(df.duration-8,47)*(df.duration>8)))",coef_Subsequent_tour_is_work_tour_Duration_lt_8_hours -util_Subsequent_tour_is_work_tour_Duration_gt_8_hours,SCHOOL - Subsequent tour is work tour: Duration > 8 hours,"@(((df.tour_count>1) & (df.tour_num == 1) & (df.tour_type.shift(-1) == 'work')) & (df.duration>8)) * ((np.minimum(8-df.duration,47)*(df.duration<=8)) + (np.minimum(df.duration-8,47)*(df.duration>8)))",coef_Subsequent_tour_is_work_tour_Duration_gt_8_hours -util_Subsequent_tour_is_school_tour_Departure_after_800_am,SCHOOL - Subsequent tour is school tour: Departure after 8:00 am,"@(((df.tour_count>1) & (df.tour_num == 1) & (df.tour_type.shift(-1) == 'school')) & (df.start>10)) *((np.minimum(10-df.start,48)*(df.start<=10)) + (np.minimum(df.start-10,48)*(df.start>10)))",coef_Subsequent_tour_is_school_tour_Departure_after_800_am -util_Subsequent_tour_is_school_tour_Duration_lt_8_hours,SCHOOL - Subsequent tour is school tour: Duration < 8 hours,"@(((df.tour_count>1) & (df.tour_num == 1) & (df.tour_type.shift(-1) == 'school')) & (df.duration<8)) * ((np.minimum(8-df.duration,47)*(df.duration<=8)) + (np.minimum(df.duration-8,47)*(df.duration>8)))",coef_Subsequent_tour_is_school_tour_Duration_lt_8_hours -util_Subsequent_tour_is_school_tour_Duration_gt_8_hours,SCHOOL - Subsequent tour is school tour: Duration > 8 hours,"@(((df.tour_count>1) & (df.tour_num == 1) & (df.tour_type.shift(-1) == 'school')) & (df.duration>8)) * ((np.minimum(8-df.duration,47)*(df.duration<=8)) + (np.minimum(df.duration-8,47)*(df.duration>8)))",coef_Subsequent_tour_is_school_tour_Duration_gt_8_hours +util_Subsequent_tour_is_work_tour_Duration_lt_8_hours,SCHOOL - Subsequent tour is work tour: Duration < 8 hours,"@(((df.tour_count>1) & (df.tour_num == 1) & (df.subsequent_tour_is_work)) & (df.duration<8)) * ((np.minimum(8-df.duration,47)*(df.duration<=8)) + (np.minimum(df.duration-8,47)*(df.duration>8)))",coef_Subsequent_tour_is_work_tour_Duration_lt_8_hours +util_Subsequent_tour_is_work_tour_Duration_gt_8_hours,SCHOOL - Subsequent tour is work tour: Duration > 8 hours,"@(((df.tour_count>1) & (df.tour_num == 1) & (df.subsequent_tour_is_work)) & (df.duration>8)) * ((np.minimum(8-df.duration,47)*(df.duration<=8)) + (np.minimum(df.duration-8,47)*(df.duration>8)))",coef_Subsequent_tour_is_work_tour_Duration_gt_8_hours +util_Subsequent_tour_is_school_tour_Departure_after_800_am,SCHOOL - Subsequent tour is school tour: Departure after 8:00 am,"@(((df.tour_count>1) & (df.tour_num == 1) & (df.subsequent_tour_is_school)) & (df.start>10)) *((np.minimum(10-df.start,48)*(df.start<=10)) + (np.minimum(df.start-10,48)*(df.start>10)))",coef_Subsequent_tour_is_school_tour_Departure_after_800_am +util_Subsequent_tour_is_school_tour_Duration_lt_8_hours,SCHOOL - Subsequent tour is school tour: Duration < 8 hours,"@(((df.tour_count>1) & (df.tour_num == 1) & (df.subsequent_tour_is_school)) & (df.duration<8)) * ((np.minimum(8-df.duration,47)*(df.duration<=8)) + (np.minimum(df.duration-8,47)*(df.duration>8)))",coef_Subsequent_tour_is_school_tour_Duration_lt_8_hours +util_Subsequent_tour_is_school_tour_Duration_gt_8_hours,SCHOOL - Subsequent tour is school tour: Duration > 8 hours,"@(((df.tour_count>1) & (df.tour_num == 1) & (df.subsequent_tour_is_school)) & (df.duration>8)) * ((np.minimum(8-df.duration,47)*(df.duration<=8)) + (np.minimum(df.duration-8,47)*(df.duration>8)))",coef_Subsequent_tour_is_school_tour_Duration_gt_8_hours util_Second_tour_of_two_mandatory_tours_Duration_lt_4_hours,SCHOOL - Second tour of two mandatory tours: Duration < 4 hours,"@(((df.tour_count>1) & (df.tour_num > 1)) & (df.duration<7)) * ((np.minimum(7-df.duration,47)*(df.duration<=7)) + (np.minimum(df.duration-7,47)*(df.duration>7)))",coef_Second_tour_of_two_mandatory_tours_Duration_lt_4_hours util_Second_tour_of_two_mandatory_tours_Duration_gt_4_hours,SCHOOL - Second tour of two mandatory tours: Duration > 4 hours,"@(((df.tour_count>1) & (df.tour_num > 1)) & (df.duration>7)) * ((np.minimum(7-df.duration,47)*(df.duration<=7)) + (np.minimum(df.duration-7,47)*(df.duration>7)))",coef_Second_tour_of_two_mandatory_tours_Duration_gt_4_hours util_Departure_Constant_Before_0600_AM,SCHOOL - Departure Constant: Before 06:00 AM,@(df.start<7),coef_Departure_Constant_Before_0600_AM @@ -56,4 +56,4 @@ util_Duration_Constant_8p5_hours,SCHOOL - Duration Constant: 8.5 hours,@(df.dura util_Duration_Constant_9_hours,SCHOOL - Duration Constant: 9 hours,@(df.duration==18),coef_Duration_Constant_9_hours util_Duration_Constant_Longer_than_9_hrs,SCHOOL - Duration Constant: Longer than 9 hrs,@(df.duration>18),coef_Duration_Constant_Longer_than_9_hrs util_Duration_Constant_Shift_for_every_30_minutes_more_than_9p5_hrs_Linear,SCHOOL - Duration Constant: Shift for every 30 minutes more than 9.5 hrs - Linear,"@(df.duration>19) * ((np.minimum(13-df.duration,47)*(df.duration<13)) + (np.minimum(df.duration-19,9)*(df.duration>19)))",coef_Duration_Constant_Shift_for_every_30_minutes_more_than_9p5_hrs_Linear -util_Duration_Constant_Shift_for_every_30_minutes_more_than_9p5_hrs_Squared,SCHOOL - Duration Constant: Shift for every 30 minutes more than 9.5 hrs - Squared,"@(df.duration>19) * (((np.minimum(13-df.duration,47)*(df.duration<13)) + (np.minimum(df.duration-19,9)*(df.duration>19))) ** 2)",coef_Duration_Constant_Shift_for_every_30_minutes_more_than_9p5_hrs_Squared +util_Duration_Constant_Shift_for_every_30_minutes_more_than_9p5_hrs_Squared,SCHOOL - Duration Constant: Shift for every 30 minutes more than 9.5 hrs - Squared,"@(df.duration>19) * (((np.minimum(13-df.duration,47)*(df.duration<13)) + (np.minimum(df.duration-19,9)*(df.duration>19))).astype(np.float32) ** 2)",coef_Duration_Constant_Shift_for_every_30_minutes_more_than_9p5_hrs_Squared diff --git a/activitysim/examples/prototype_mwcog/configs/trip_destination.csv b/activitysim/examples/prototype_mwcog/configs/trip_destination.csv index 3f950aa96b..e3c9558b76 100644 --- a/activitysim/examples/prototype_mwcog/configs/trip_destination.csv +++ b/activitysim/examples/prototype_mwcog/configs/trip_destination.csv @@ -1,6 +1,6 @@ Label,Description,Expression,work,univ,school,escort,shopping,eatout,othmaint,social,othdiscr,atwork -util_size_term,size term,"@np.log1p(size_terms.get(df.dest_taz, df.purpose))",coef_one,coef_one,coef_one,coef_one,coef_one,coef_one,coef_one,coef_one,coef_one,coef_one -util_no_attractions,no attractions,"@size_terms.get(df.dest_taz, df.purpose) == 0",coef_unavailable,coef_unavailable,coef_unavailable,coef_unavailable,coef_unavailable,coef_unavailable,coef_unavailable,coef_unavailable,coef_unavailable,coef_unavailable +util_size_term,size term,"@np.log1p(size_terms.get(df.dest_taz, df.purpose)) # sharrow: np.log1p(size_terms['arry'])",coef_one,coef_one,coef_one,coef_one,coef_one,coef_one,coef_one,coef_one,coef_one,coef_one +util_no_attractions,no attractions,"@size_terms.get(df.dest_taz, df.purpose) == 0 # sharrow: size_terms['arry'] == 0",coef_unavailable,coef_unavailable,coef_unavailable,coef_unavailable,coef_unavailable,coef_unavailable,coef_unavailable,coef_unavailable,coef_unavailable,coef_unavailable #util_stop_zone_CDB_are_type,#stop zone CBD area type,"@reindex(land_use.area_type, df.dest_taz) < setting('cbd_threshold')",,,,,,,,,, util_distance_inbound,distance (calibration adjustment individual - inbound),@(~df.is_joint & ~df.outbound) * (od_skims['DIST'] + dp_skims['DIST']),coef_util_distance_work_outbound,coef_util_distance_univ,coef_util_distance_school,coef_util_distance_escort,coef_util_distance_shopping,coef_util_distance_eatout,coef_util_distance_othmaint,coef_util_distance_social,coef_util_distance_othdiscr,coef_util_distance_atwork util_distance_outbound,distance (calibration adjustment individual - outbound),@(~df.is_joint & df.outbound) * (od_skims['DIST'] + dp_skims['DIST']),coef_util_distance_work_inbound,coef_util_distance_univ,coef_util_distance_school,coef_util_distance_escort,coef_util_distance_shopping,coef_util_distance_eatout,coef_util_distance_othmaint,coef_util_distance_social,coef_util_distance_othdiscr,coef_util_distance_atwork diff --git a/activitysim/examples/prototype_mwcog/configs/trip_destination_annotate_trips_preprocessor.csv b/activitysim/examples/prototype_mwcog/configs/trip_destination_annotate_trips_preprocessor.csv index 0217a68555..3f60fe0175 100644 --- a/activitysim/examples/prototype_mwcog/configs/trip_destination_annotate_trips_preprocessor.csv +++ b/activitysim/examples/prototype_mwcog/configs/trip_destination_annotate_trips_preprocessor.csv @@ -7,3 +7,4 @@ Description,Target,Expression #,,not needed as school is not chosen as an intermediate trip destination #,_grade_school,"(df.primary_purpose == 'school') & reindex(persons.is_gradeschool, df.person_id)" #,size_segment,"df.primary_purpose.where(df.primary_purpose != 'school', np.where(_grade_school,'gradeschool', 'highschool'))" +,purpose_index_num,"size_terms.get_cols(df.purpose)" diff --git a/activitysim/examples/prototype_mwcog/configs/trip_destination_sample.csv b/activitysim/examples/prototype_mwcog/configs/trip_destination_sample.csv index 6a0a53480e..0b1253684f 100644 --- a/activitysim/examples/prototype_mwcog/configs/trip_destination_sample.csv +++ b/activitysim/examples/prototype_mwcog/configs/trip_destination_sample.csv @@ -6,8 +6,8 @@ Not available if walk tour not within walking distance,@(df.tour_mode=='WALK') & Not available if bike tour not within biking distance,@(df.tour_mode=='BIKE') & (od_skims['DISTBIKE'] > max_bike_distance),-999,-999,-999,-999,-999,-999,-999,-999,-999,-999 Not available if bike tour not within biking distance,@(df.tour_mode=='BIKE') & (dp_skims['DISTBIKE'] > max_bike_distance),-999,-999,-999,-999,-999,-999,-999,-999,-999,-999 #If transit tour is not in walk sub-zone it must be walkable,,,,,,,,,,, -size term,"@np.log1p(size_terms.get(df.dest_taz, df.purpose))",1,1,1,1,1,1,1,1,1,1 -no attractions,"@size_terms.get(df.dest_taz, df.purpose) == 0",-999,-999,-999,-999,-999,-999,-999,-999,-999,-999 +size term,"@np.log1p(size_terms.get(df.dest_taz, df.purpose)) # sharrow: np.log1p(size_terms['arry'])",1,1,1,1,1,1,1,1,1,1 +no attractions,"@size_terms.get(df.dest_taz, df.purpose) == 0 # sharrow: size_terms['arry'] == 0",-999,-999,-999,-999,-999,-999,-999,-999,-999,-999 #stop zone CBD area type,"@reindex(land_use.AreaType, df.dest_taz) < setting('cbd_threshold')",,,,,,,,,, distance (calibration adjustment individual - inbound),@(~df.is_joint & ~df.outbound) * (_od_DIST + _dp_DIST),-0.049725916,-0.0613,-0.1056,-0.1491,-0.1192,-0.1029,-0.0962,-0.1329,-0.126172224,-0.122334597 distance (calibration adjustment individual - outbound),@(~df.is_joint & df.outbound) * (_od_DIST + _dp_DIST),0.147813279,-0.0613,-0.1056,-0.1491,-0.1192,-0.1029,-0.0962,-0.1329,-0.126172224,-0.122334597 diff --git a/activitysim/examples/prototype_mwcog/configs/trip_mode_choice.csv b/activitysim/examples/prototype_mwcog/configs/trip_mode_choice.csv index fa8def35f6..0ce978b1e3 100644 --- a/activitysim/examples/prototype_mwcog/configs/trip_mode_choice.csv +++ b/activitysim/examples/prototype_mwcog/configs/trip_mode_choice.csv @@ -8,7 +8,7 @@ util_DRIVEALONE_In_vehicle_time,DRIVEALONE - In-vehicle time,@odt_skims['SOV_TIM util_DRIVEALONE_Terminal_time,DRIVEALONE - Terminal time,@coef_walktimeshort_multiplier * df.total_terminal_time,coef_ivt,,,,,,,,,,,,,,,,,,,, util_DRIVEALONE_Operating_cost_,DRIVEALONE - Operating cost ,@ivt_cost_multiplier * df.ivot * costPerMile * odt_skims['SOV_DIST'],coef_ivt,,,,,,,,,,,,,,,,,,,, util_DRIVEALONE_Parking_cost_,DRIVEALONE - Parking cost ,@ivt_cost_multiplier * df.ivot * df.total_parking_cost,coef_ivt,,,,,,,,,,,,,,,,,,,, -util_DRIVEALONE_Person_is_between_16_and_19_years_old,DRIVEALONE - Person is between 16 and 19 years old,@c_age1619_da * ((df.age >= 16) & (df.age <= 19)),coef_age1619_da,,,,,,,,,,,,,,,,,,,, +#util_DRIVEALONE_Person_is_between_16_and_19_years_old,DRIVEALONE - Person is between 16 and 19 years old,@c_age1619_da * ((df.age >= 16) & (df.age <= 19)),coef_age1619_da,,,,,,,,,,,,,,,,,,,, #Shared_ride_2,#Shared ride 2,,,,,,,,,,,,,,,,,,,,,, util_SHARED2_Unavailable,SHARED2 - Unavailable,hov2_available == False,,coef_unavailable,,,,,,,,,,,,,,,,,,, util_SHARED2_Unavailable_based_on_party_size,SHARED2 - Unavailable based on party size,is_joint & (number_of_participants > 2),,coef_unavailable,,,,,,,,,,,,,,,,,,, @@ -18,7 +18,7 @@ util_SHARED2_Operating_cost,SHARED2 - Operating cost,@ivt_cost_multiplier * df.i util_SHARED2_Parking_cost,SHARED2 - Parking cost,@ivt_cost_multiplier * df.ivot * df.total_parking_cost / costShareSr2,,coef_ivt,,,,,,,,,,,,,,,,,,, util_SHARED2_One_person_household,SHARED2 - One person household,@(df.hhsize == 1),,coef_hhsize1_sr,,,,,,,,,,,,,,,,,,, util_SHARED2_Two_person_household,SHARED2 - Two person household,@(df.hhsize == 2),,coef_hhsize2_sr,,,,,,,,,,,,,,,,,,, -util_SHARED2_Person_is_16_years_old_or_older,SHARED2 - Person is 16 years old or older,@(df.age >= 16),,coef_age1619_da,,,,,,,,,,,,,,,,,,, +#util_SHARED2_Person_is_16_years_old_or_older,SHARED2 - Person is 16 years old or older,@(df.age >= 16),,coef_age1619_da,,,,,,,,,,,,,,,,,,, #Shared_ride_3+,#Shared ride 3+,,,,,,,,,,,,,,,,,,,,,, util_SHARED3_Unavailable,SHARED3 - Unavailable,hov3_available == False,,,coef_unavailable,,,,,,,,,,,,,,,,,, util_SHARED3_Unavailable_based_joint_tour_mode,SHARED3 - Unavailable based joint tour mode,@df.is_joint & df.i_tour_mode.isin(I_SR2_MODES),,,coef_unavailable,,,,,,,,,,,,,,,,,, diff --git a/activitysim/examples/prototype_mwcog/configs/work_from_home.csv b/activitysim/examples/prototype_mwcog/configs/work_from_home.csv index b6f9853df8..fed5611bca 100644 --- a/activitysim/examples/prototype_mwcog/configs/work_from_home.csv +++ b/activitysim/examples/prototype_mwcog/configs/work_from_home.csv @@ -2,9 +2,9 @@ Label,Description,Expression,work_at_home,work_away_from_home util_work_from_home_constant,Constant for Working from home,1,coef_work_from_home_constant,0 util_full_time_worker,Full time worker (1 if true),@df.pemploy==PEMPLOY_FULL,coef_full_time_worker,0 util_female_worker,Female Worker,@df.SEX==2,coef_female_worker,0 -util_female_worker_preschool_child,Female worker with a Preschool Child in Household,"@df.SEX==2 & other_than(df.household_id, df.ptype == PTYPE_SCHOOL)",coef_female_worker_preschool_child,0 +util_female_worker_preschool_child,Female worker with a Preschool Child in Household,female_worker_with_preschool_child,coef_female_worker_preschool_child,0 util_access_to_workplaces,Accessibility to workplaces of the home mgra,@df.workplace_location_logsum,coef_access_to_workplaces,0 -util_non_working_adult_in_hh,Presence of Non Working Adult in the Household,"@other_than(df.household_id, df.ptype == PTYPE_NONWORK)",coef_non_working_adult_in_hh,0 +util_non_working_adult_in_hh,Presence of Non Working Adult in the Household,nonworking_adult_in_hh,coef_non_working_adult_in_hh,0 util_education_ba_plus,Education Level Bachelors or higher degree,0,coef_education_ba_plus,0 util_low_income,Household income Less than 30K,@df.income < 30000,coef_low_income,0 util_age_lt_35,Age Group - Less than 35 years,@df.age < 35,coef_age_lt_35,0 diff --git a/activitysim/examples/prototype_mwcog/configs/work_from_home.yaml b/activitysim/examples/prototype_mwcog/configs/work_from_home.yaml index c4f265ced9..8938761450 100644 --- a/activitysim/examples/prototype_mwcog/configs/work_from_home.yaml +++ b/activitysim/examples/prototype_mwcog/configs/work_from_home.yaml @@ -10,3 +10,7 @@ LOGIT_TYPE: MNL WORK_FROM_HOME_ALT: 0 DEST_CHOICE_COLUMN_NAME: workplace_zone_id + +preprocessor: + SPEC: work_from_home_annotate_persons_preprocessor + DF: persons diff --git a/activitysim/examples/prototype_mwcog/configs/work_from_home_annotate_persons_preprocessor.csv b/activitysim/examples/prototype_mwcog/configs/work_from_home_annotate_persons_preprocessor.csv new file mode 100644 index 0000000000..299f422f72 --- /dev/null +++ b/activitysim/examples/prototype_mwcog/configs/work_from_home_annotate_persons_preprocessor.csv @@ -0,0 +1,3 @@ +Description,Target,Expression +,female_worker_with_preschool_child,"df.SEX==2 & other_than(df.household_id, df.ptype == PTYPE_SCHOOL)" +,nonworking_adult_in_hh,"other_than(df.household_id, df.ptype == PTYPE_NONWORK)" diff --git a/activitysim/examples/prototype_mwcog/configs_sharrow/network_los.yaml b/activitysim/examples/prototype_mwcog/configs_sharrow/network_los.yaml new file mode 100644 index 0000000000..1c4cd79daf --- /dev/null +++ b/activitysim/examples/prototype_mwcog/configs_sharrow/network_los.yaml @@ -0,0 +1,6 @@ +inherit_settings: True + +# read cached skims (using numpy memmap) from output directory (memmap is faster than omx ) +read_skim_cache: False +# write memmapped cached skims to output directory after reading from omx, for use in subsequent runs +write_skim_cache: False diff --git a/activitysim/examples/prototype_mwcog/configs_sharrow/settings.yaml b/activitysim/examples/prototype_mwcog/configs_sharrow/settings.yaml new file mode 100644 index 0000000000..7ddeaad9cf --- /dev/null +++ b/activitysim/examples/prototype_mwcog/configs_sharrow/settings.yaml @@ -0,0 +1,3 @@ +inherit_settings: True +sharrow: require +recode_pipeline_columns: True diff --git a/activitysim/examples/prototype_mwcog/extensions/telecommute_frequency.py b/activitysim/examples/prototype_mwcog/extensions/telecommute_frequency.py index c370814910..a299d2832c 100644 --- a/activitysim/examples/prototype_mwcog/extensions/telecommute_frequency.py +++ b/activitysim/examples/prototype_mwcog/extensions/telecommute_frequency.py @@ -4,19 +4,13 @@ import pandas as pd -from activitysim.core import tracing -from activitysim.core import config -from activitysim.core import pipeline -from activitysim.core import simulate -from activitysim.core import inject -from activitysim.core import expressions - from activitysim.abm.models.util import estimation +from activitysim.core import config, expressions, inject, pipeline, simulate, tracing logger = logging.getLogger(__name__) -@inject.step() +@inject.custom_step() def telecommute_frequency(persons_merged, persons, chunk_size, trace_hh_id): """ This model predicts the frequency of telecommute for a person (worker) who diff --git a/activitysim/examples/prototype_mwcog/extensions/transit_pass_ownership.py b/activitysim/examples/prototype_mwcog/extensions/transit_pass_ownership.py index ebe43953f3..e2d020443c 100644 --- a/activitysim/examples/prototype_mwcog/extensions/transit_pass_ownership.py +++ b/activitysim/examples/prototype_mwcog/extensions/transit_pass_ownership.py @@ -2,21 +2,13 @@ # See full license in LICENSE.txt. import logging -import numpy as np - -from activitysim.core import tracing -from activitysim.core import config -from activitysim.core import pipeline -from activitysim.core import simulate -from activitysim.core import inject -from activitysim.core import expressions - from activitysim.abm.models.util import estimation +from activitysim.core import config, expressions, inject, pipeline, simulate, tracing logger = logging.getLogger("activitysim") -@inject.step() +@inject.custom_step() def transit_pass_ownership(persons_merged, persons, chunk_size, trace_hh_id): """ Transit pass ownership model. diff --git a/activitysim/examples/prototype_mwcog/extensions/transit_pass_subsidy.py b/activitysim/examples/prototype_mwcog/extensions/transit_pass_subsidy.py index e902753cb3..7a4408fe35 100644 --- a/activitysim/examples/prototype_mwcog/extensions/transit_pass_subsidy.py +++ b/activitysim/examples/prototype_mwcog/extensions/transit_pass_subsidy.py @@ -2,21 +2,13 @@ # See full license in LICENSE.txt. import logging -import numpy as np - -from activitysim.core import tracing -from activitysim.core import config -from activitysim.core import pipeline -from activitysim.core import simulate -from activitysim.core import inject -from activitysim.core import expressions - from activitysim.abm.models.util import estimation +from activitysim.core import config, expressions, inject, pipeline, simulate, tracing logger = logging.getLogger("activitysim") -@inject.step() +@inject.custom_step() def transit_pass_subsidy(persons_merged, persons, chunk_size, trace_hh_id): """ Transit pass subsidy model. diff --git a/activitysim/examples/prototype_mwcog/extensions/work_from_home.py b/activitysim/examples/prototype_mwcog/extensions/work_from_home.py index 82d6618d65..0cc032ea78 100644 --- a/activitysim/examples/prototype_mwcog/extensions/work_from_home.py +++ b/activitysim/examples/prototype_mwcog/extensions/work_from_home.py @@ -4,19 +4,13 @@ import numpy as np -from activitysim.core import tracing -from activitysim.core import config -from activitysim.core import pipeline -from activitysim.core import simulate -from activitysim.core import inject -from activitysim.core import expressions - from activitysim.abm.models.util import estimation +from activitysim.core import config, expressions, inject, pipeline, simulate, tracing logger = logging.getLogger(__name__) -@inject.step() +@inject.custom_step() def work_from_home(persons_merged, persons, chunk_size, trace_hh_id): """ This model predicts the whether a person (worker) works from home. The output diff --git a/activitysim/examples/prototype_mwcog/simulation.py b/activitysim/examples/prototype_mwcog/simulation.py index 3126777b84..b5cd37ea8d 100644 --- a/activitysim/examples/prototype_mwcog/simulation.py +++ b/activitysim/examples/prototype_mwcog/simulation.py @@ -1,12 +1,12 @@ # ActivitySim # See full license in LICENSE.txt. -import sys import argparse +import sys -from activitysim.cli.run import add_run_args, run +import extensions # noqa: F401 -import extensions +from activitysim.cli.run import add_run_args, run if __name__ == "__main__": diff --git a/activitysim/examples/prototype_mwcog/test/configs/settings.yaml b/activitysim/examples/prototype_mwcog/test/configs/settings.yaml index 12820a3412..b9597e83aa 100644 --- a/activitysim/examples/prototype_mwcog/test/configs/settings.yaml +++ b/activitysim/examples/prototype_mwcog/test/configs/settings.yaml @@ -24,16 +24,8 @@ want_dest_choice_presampling: True # - tracing # trace household id; comment out or leave empty for no trace -trace_hh_id: +#trace_hh_id: # trace origin, destination in accessibility calculation; comment out or leave empty for no trace # trace_od: [5, 11] trace_od: - -output_tables: - h5_store: False - action: include - prefix: final_ - sort: True - tables: - - trips diff --git a/activitysim/examples/prototype_mwcog/test/regress/final_accessibility.csv b/activitysim/examples/prototype_mwcog/test/regress/final_accessibility.csv new file mode 100644 index 0000000000..b2580500aa --- /dev/null +++ b/activitysim/examples/prototype_mwcog/test/regress/final_accessibility.csv @@ -0,0 +1,54 @@ +zone_id,auPkRetail,auPkTotal,auOpRetail,auOpTotal,trPkRetail,trPkTotal,trOpRetail,trOpTotal,nmRetail,nmTotal +1,9.81029707051306,12.151784326376733,10.255286731654158,12.56149659149444,6.876715840695639,9.397924315741278,6.053229178214222,8.480448597281763,7.862428648487154,10.55193711294014 +2,9.788277306830071,12.132040532912322,10.214609253692055,12.52197012399085,6.037116576669806,8.690508031671603,4.980478571350719,7.992674074446341,7.084710260664416,9.826611142334702 +7,9.951414580650491,12.283099685288235,10.240738778496745,12.548006556801958,6.4688768820585985,9.018113778183793,4.960877821706156,7.887901384870883,7.873290248812872,10.496318027463456 +8,10.030356609848026,12.352121210064473,10.291416910116402,12.595211901215755,6.738637280075503,9.182418618904922,6.452755385876383,8.789757913062887,8.214553500152073,10.716177880569457 +9,10.020962632947493,12.34903048751262,10.275835312062787,12.581812048948786,6.124123330603821,8.644302265895908,5.3676452956407825,7.913964536687555,8.128433507557613,10.654159711505082 +10,9.763317308325297,12.09944514868773,10.218295122561322,12.525639944917279,5.904437143418533,8.358141088934934,5.771143227101617,8.156395157757155,7.332279313228723,9.98445175585946 +11,9.778268143705438,12.117164584933347,10.203218473646277,12.512926752956862,6.467227018619716,8.776753419086738,6.438986190595064,8.768992361563054,7.516702730950308,10.185183992585701 +12,9.778273071167837,12.117396279188348,10.209945608968326,12.519979857557024,7.398700074392646,9.670430176801533,7.362552427398613,9.64330155302293,7.478455753872086,10.187418893586308 +13,10.041556578649663,12.365979380652888,10.2843844014615,12.590339207788306,6.818691837353038,9.233062819644022,6.8240347945859705,9.227808322232034,8.371176716362479,10.878363745564682 +14,10.079216962899368,12.387808431741998,10.290650973736224,12.591631453841016,7.593535571333513,9.889078458983404,7.5350704183973525,9.890098037989627,8.621595050570889,11.016431809450307 +15,9.976460948049983,12.302571517613107,10.245705084038816,12.552675997490658,7.18309523631608,9.5093225150168,7.135296922854232,9.449220363139727,8.095069087912316,10.667693152052896 +16,9.967253096020036,12.282129528352327,10.257759877405393,12.560152966945482,7.439501338969431,9.668075148412353,7.10674599988696,9.42424286613223,8.268391091588258,10.710886474825353 +17,9.84710926015887,12.154456202826767,10.196692007352805,12.495970359636935,6.421665187609401,8.747643104793209,6.232826025643736,8.521774155299333,8.041387852192658,10.34712925494038 +18,9.8809319103354,12.17586916769478,10.231303777505538,12.524272953433846,7.62450174369167,9.85900139759549,7.438030883855165,9.660774576309322,8.338462854992917,10.558523314369511 +19,9.989627843419285,12.28907532740037,10.257308996922223,12.551328746256361,7.334960948483767,9.60289221653445,7.072882849537048,9.351978969326513,8.487768811478443,10.75734128351947 +20,10.032712518459942,12.317236757322984,10.26263851104096,12.549187139572146,7.293153312763276,9.588183876671874,7.1039492999445555,9.38630919558874,8.56928567801849,10.759140866243634 +21,10.146496106980027,12.446639939386053,10.314421942913885,12.609749571700796,7.749896258110251,9.91690446721725,7.565577801087948,9.714059315540243,8.91407161586761,11.245506636525104 +22,10.136504165772347,12.443572051131406,10.31632105589481,12.615821477393153,7.384782482520821,9.66826286324278,7.09364594338033,9.258656483098955,8.833215152900111,11.20225326022628 +23,10.219141527027782,12.505907175669392,10.348512339595269,12.634740346986396,8.089863512282314,10.35403983209601,7.983455788985605,10.25879479069529,9.156416158648906,11.398422792872404 +24,10.189506840714003,12.478639501945677,10.325513115167698,12.61248848295675,8.058929624437695,10.320368483898426,7.950099495849735,10.219980550758034,9.011350548570737,11.271186348085068 +25,10.176994412820521,12.454085300515493,10.334780647390623,12.614579085966184,7.326336436121433,9.60634186736448,7.207516305510279,9.492889659881886,9.007523827897002,11.207796515240538 +26,10.25392071248298,12.531917000277199,10.367179157087357,12.647865075118984,7.7791800664779105,9.92704868966743,7.506808028172236,9.759083498837878,9.236679911415154,11.455065824866923 +27,10.236547173428612,12.507728766116474,10.360825168984713,12.636715193644552,7.578440285480581,9.825382103561257,7.5590679838630175,9.805911145866416,9.125560574736374,11.320923987364303 +28,10.301745624472627,12.581174996300112,10.377601157005566,12.657929212057196,7.574637557553662,9.871871369150544,7.503620555855803,9.770628826056212,9.281431527984184,11.52644763343522 +29,10.261190883971448,12.544137410762088,10.373788191379772,12.657609719426917,7.989062965994412,10.265232043719008,7.96961987193324,10.222842478287195,9.183222808115536,11.415962302034643 +30,10.193067981446358,12.486575463630118,10.344118809810192,12.632591609388577,7.272757546070672,9.797945378328468,6.954091370472409,9.48717261415758,9.066551175212254,11.334567581207724 +31,10.14821695396716,12.456044739791448,10.328275481225049,12.626997020363897,7.683754338868759,9.991163186394598,7.181588961266754,9.461152791996348,8.775134929221027,11.155362487186597 +32,10.199676503076857,12.486316055706531,10.324945894734693,12.607901620887345,7.151672543145741,9.42178487641376,7.04074217327661,9.314118746526368,8.802177621489955,11.075585426434099 +33,10.109843818280945,12.4234447990138,10.318545809769663,12.61981171371549,7.553950338860209,9.901284253403565,6.149998002147215,8.472813900557007,8.379225042349733,10.898050926528375 +34,10.075011160954372,12.391130331598925,10.306865380188055,12.608971075465956,7.826213977495006,10.249543185916705,6.013864036661982,8.546484096081763,8.24101023562675,10.79160511644648 +35,10.14799500837615,12.448792671302938,10.321844853197955,12.615953083442333,6.711857565558342,9.359050010644143,6.3917991199339275,8.755226326453794,8.68090664300488,11.060497130450878 +36,10.196210537159804,12.492757272878901,10.348975266099023,12.63996774726812,7.543309342020908,9.841154137097512,7.247952393199388,9.501465917414611,8.961073717035672,11.286228147716592 +37,10.273346741224698,12.553974502573022,10.369986185833797,12.651345490139049,7.80391990056557,10.105767214149342,7.639930446123892,9.895230620729473,9.184883204591207,11.443060329974774 +38,10.267426804114043,12.541123514658343,10.354131529664407,12.630591887053097,7.590491040034944,9.864828930267816,7.431380108356087,9.668444949872097,9.208074528941069,11.44430434879604 +39,10.319703320147537,12.59410781067339,10.380628096028003,12.65662169611359,7.554467463879129,9.81426054941184,7.402562116340096,9.640562418099982,9.33279724613972,11.566187172053397 +40,10.313903644924546,12.584558161316187,10.373981119429155,12.649251460983688,7.283166887986826,9.554389307234306,7.154746608079152,9.38065611565189,9.271075899688343,11.46682944998638 +41,10.21664542985719,12.476012218831512,10.340741581248306,12.609500783418065,7.2627727050661255,9.564154822141253,7.094372088140492,9.412284403083117,8.979454102449493,11.121058865400364 +42,10.104040913247243,12.349339725985713,10.306040543803691,12.570920029755223,7.238095533690809,9.437168836561312,6.9857172717383325,9.175520611647908,8.830086713291724,10.92498187513246 +43,10.269541605942377,12.534502531745929,10.351796595215296,12.622769757266996,7.114872822591869,9.376728651108559,6.93847195496077,9.182215371351141,9.069917583397613,11.244133247627502 +44,10.198100825791958,12.460695355643093,10.33040572289756,12.601944415829093,7.4347891910273205,9.680041748521075,7.205568162668223,9.38466023783227,9.072186429715115,11.242187549092566 +45,10.07287697502762,12.328907161156666,10.312037962406624,12.580861685802951,7.720499447774655,9.95161344751897,7.4134064376622435,9.647247938560254,8.762652110965647,10.867212942284684 +46,9.831234474764646,12.060349879931556,10.264936240009211,12.52717460654812,7.200202134441105,9.315353067285672,6.89362783082587,9.007761001217217,8.513604821367373,10.5018950449192 +47,9.950458094488111,12.184820405874586,10.261666682051988,12.522519490342884,6.987874297729277,9.192996534299514,6.7579970754555045,8.947448247713291,8.489563622896995,10.444213037509368 +49,10.08528933318216,12.343308566514107,10.284679270232289,12.55351001810869,7.2090013681168355,9.540996232743087,6.95913918207846,9.19981333374713,8.738279917779282,10.852392869995683 +50,9.98298564252219,12.246919838111948,10.253344155919187,12.525647346980485,6.3077424436278875,8.725080889453865,6.316568981491454,8.661959196250848,8.388933539152017,10.505322717157444 +53,10.025873420322183,12.324661148540837,10.276050326657494,12.56595884543768,6.69731843293379,9.116941333573378,6.8130803690156085,9.156353178383066,8.291853314136263,10.603312277057631 +54,10.079290293777424,12.36002964504986,10.306536192127197,12.586603133829211,7.383191905630194,9.735024400590504,7.464188174097621,9.721180334726407,8.608723960438455,10.840011492374432 +55,10.161364324091105,12.440332184715814,10.31953875226402,12.599968378796609,7.299831492301443,9.538446402383238,6.999620210392883,9.257463655642184,8.881092323610316,11.12270140365654 +56,10.186456384638735,12.473570255176272,10.342179445501733,12.627228333370054,7.961298459211637,10.357448770921671,7.815139975567722,10.146148436251764,9.010981361388396,11.29370168219867 +57,10.084721079625728,12.382481809070438,10.293632257212424,12.584791972974568,7.725709061763513,10.123632623826373,7.675992514912229,9.98850035457405,8.505300859619034,10.833955915950408 +58,10.092097258305321,12.394915671728558,10.299733848685115,12.595875492008648,7.3961190589994645,9.882779126312077,7.234517509769107,9.599306370167394,8.55295658075461,10.912773536086048 +59,9.950127783917965,12.271124107664363,10.229447190037053,12.534134686077843,6.583638500449655,9.02101651098535,6.299506264639063,8.737367157725654,7.714388290690883,10.37918155440571 +60,9.730208398357577,12.048085220053384,10.228314678938698,12.524380571684402,5.688939453315608,8.095383443040733,5.507622605448901,7.856293893786833,7.340796449563834,9.744752790735387 diff --git a/activitysim/examples/prototype_mwcog/test/regress/final_households.csv b/activitysim/examples/prototype_mwcog/test/regress/final_households.csv new file mode 100644 index 0000000000..4d6a758f19 --- /dev/null +++ b/activitysim/examples/prototype_mwcog/test/regress/final_households.csv @@ -0,0 +1,11 @@ +household_id,puma_geoid,home_zone_id,TYPE,hhsize,auto_ownership,HHT,hhincadj,workers,has_children,sample_rate,income,income_in_thousands,income_segment,median_value_of_time,hh_value_of_time,num_workers,num_non_workers,num_drivers,num_adults,num_children,num_young_children,num_children_5_to_15,num_children_6_to_12,num_children_16_to_17,num_college_age,num_young_adults,non_family,family,home_is_urban,home_is_rural,TAZ,num_predrive_child,num_nonworker_adults,num_fullTime_workers,num_partTime_workers,retired_adults_only_hh,hh_work_auto_savings_ratio,num_under16_not_at_school,num_travel_active,num_travel_active_adults,num_travel_active_preschoolers,num_travel_active_children,num_travel_active_non_preschoolers,participates_in_jtf_model,joint_tour_frequency,num_hh_joint_tours +2704736,1100105,36,3,1,0,4.0,20021,1,-9,0.0,20021,20.021,1,6.01,5.707035026149024,1,0,1,1,0,0,0,0,0,0,0,True,False,True,False,36,0,0,1,0,False,0.117,0,0,0,0,0,0,False,0_tours,0 +1632,1100105,25,1,3,2,1.0,107067,1,1,0.0,107067,107.067,3,10.44,9.177177574953367,1,2,2,2,1,1,0,0,0,0,2,False,True,True,False,25,0,0,1,0,False,0.08516667,1,3,2,1,1,2,True,0_tours,0 +12682,1100105,47,1,1,1,4.0,62099,1,0,0.0,62099,62.099,2,8.81,21.56089619468371,1,0,1,1,0,0,0,0,0,0,1,True,False,True,False,47,0,0,1,0,False,0.54675,0,1,1,0,0,1,False,0_tours,0 +1590,1100105,25,1,9,1,2.0,17192,2,1,0.0,17192,17.192,1,6.01,2.1180144126507763,2,7,2,2,7,4,3,3,0,0,2,False,True,True,False,25,3,0,1,1,False,0.708,4,3,0,2,3,1,True,0_tours,0 +19840,1100105,57,1,1,1,4.0,95289,1,0,0.0,95289,95.289,2,8.81,16.892032842005793,1,0,1,1,0,0,0,0,0,0,1,True,False,True,False,57,0,0,1,0,False,0.294,0,1,1,0,0,1,False,0_tours,0 +10473,1100105,44,1,1,0,6.0,47109,1,0,0.0,47109,47.109,1,6.01,4.307904952573323,1,0,1,1,0,0,0,0,0,0,1,True,False,True,False,44,0,0,1,0,False,0.17375,0,0,0,0,0,0,False,0_tours,0 +8484,1100105,42,1,1,1,4.0,175104,1,0,0.0,175104,175.104,4,12.86,14.589903621439268,1,0,1,1,0,0,0,0,0,0,1,True,False,True,False,42,0,0,1,0,False,0.50116664,0,1,1,0,0,1,False,0_tours,0 +11636,1100105,47,1,2,2,5.0,265310,2,0,0.0,265310,265.31,4,12.86,10.475855530295568,2,0,2,2,0,0,0,0,0,0,1,True,False,True,False,47,0,0,2,0,False,0.6155,0,2,2,0,0,2,True,1_Shop,1 +14281,1100105,49,1,1,1,4.0,73804,1,0,0.0,73804,73.804,2,8.81,9.579868420839766,1,0,1,1,0,0,0,0,0,0,0,True,False,True,False,49,0,0,1,0,False,0.50525004,0,1,1,0,0,1,False,0_tours,0 +1620,1100105,25,1,3,1,5.0,168841,2,0,0.0,168841,168.841,4,12.86,8.070805737817167,2,1,3,3,0,0,0,0,0,0,3,True,False,True,False,25,0,1,2,0,False,0.4055,0,3,3,0,0,3,True,0_tours,0 diff --git a/activitysim/examples/prototype_mwcog/test/regress/final_joint_tour_participants.csv b/activitysim/examples/prototype_mwcog/test/regress/final_joint_tour_participants.csv new file mode 100644 index 0000000000..2ff6bba455 --- /dev/null +++ b/activitysim/examples/prototype_mwcog/test/regress/final_joint_tour_participants.csv @@ -0,0 +1,3 @@ +participant_id,tour_id,household_id,person_id,participant_num +4770766001,47707660,11636,1163601,1 +4770766002,47707660,11636,1163602,2 diff --git a/activitysim/examples/prototype_mwcog/test/regress/final_land_use.csv b/activitysim/examples/prototype_mwcog/test/regress/final_land_use.csv new file mode 100644 index 0000000000..4ec3dec73e --- /dev/null +++ b/activitysim/examples/prototype_mwcog/test/regress/final_land_use.csv @@ -0,0 +1,54 @@ +zone_id,HH,HHPOP,GQPOP,TOTPOP,TOTEMP,INDEMP,RETEMP,OFFEMP,OTHEMP,JURCODE,LANDAREA,HHINCIDX,ADISTTOX,TAZXCRD,TAZYCRD,K_8,G9_12,COLLEGE,Park_Acres,GC_Acres,PRKCST,OPRKCST,TERMINAL,AREATYPE,household_density,employment_density,density_index,TOPOLOGY +1,0,0,0,0,12623,29,149,11731,714,0,0.0424,10,29.08,1298543,446898,0,0,0,2.906924,0.0,117.875,200,5,1,0.0,465.1754127358491,0.0,1 +2,0,0,0,0,0,0,0,0,0,0,0.1553,5,29.28,1298807,445281,0,0,0,106.708067,0.0,103.75,200,5,1,0.0,0.0,0.0,1 +7,0,0,0,0,0,0,0,0,0,0,0.0814,10,29.09,1299596,445915,0,0,0,51.717837,0.0,116.125,200,5,1,0.0,0.0,0.0,1 +8,0,0,0,0,0,0,0,0,0,0,0.0708,10,28.68,1301916,446878,0,0,0,42.722665,0.0,126.5,200,5,1,0.0,0.0,0.0,1 +9,0,0,0,0,837,32,32,679,94,0,0.1368,10,28.9,1302004,445336,0,0,0,91.542163,0.0,118.75,200,5,1,0.0,9.56003289473684,0.0,1 +10,0,0,0,0,40,4,4,28,4,0,0.0585,10,29.03,1302622,443982,0,0,0,24.899056,0.0,114.625,100,4,2,0.0,1.0683760683760681,0.0,1 +11,77,118,0,118,11123,153,690,10237,43,0,0.0965,10,28.92,1303826,443797,0,0,0,4.678476,0.0,113.375,200,5,1,1.246761658031088,180.10038860103626,1.2381901716321244,1 +12,0,0,0,0,8674,45,80,8410,140,0,0.0645,10,28.71,1305207,444137,0,0,0,4.58145,0.0,113.5,200,5,1,0.0,210.12596899224806,0.0,1 +13,0,0,0,0,0,0,0,0,0,0,0.0502,10,28.64,1303781,445659,0,0,0,29.514458,0.0,121.375,200,5,1,0.0,0.0,0.0,1 +14,0,0,0,0,3061,28,65,2868,101,0,0.0366,10,28.36,1304865,446730,0,0,0,1.153893,0.0,121.5,200,5,1,0.0,130.67793715846994,0.0,1 +15,0,0,0,0,2552,13,78,1923,538,0,0.1033,10,28.51,1305222,445451,17,0,0,62.05587,0.0,118.125,200,5,1,0.0,38.60116166505325,0.0,1 +16,0,0,11,11,1151,2,1,1134,14,0,0.021,10,28.26,1306024,446507,0,0,0,1.927624,0.0,117.5,200,5,1,0.0,85.63988095238095,0.0,1 +17,94,123,0,123,8168,32,557,7065,515,0,0.0604,10,28.04,1307606,446777,0,0,0,4.556058999999999,0.0,118.75,200,5,1,2.431705298013245,211.2996688741722,2.4040388373483643,1 +18,2,9,165,174,2753,18,153,2510,73,0,0.0501,3,27.93,1307404,447663,0,0,0,12.798207,0.0,119.25,200,5,1,0.062375249500998,85.85953093812375,0.06232996801315698,1 +19,935,1370,0,1370,4952,113,782,3743,314,0,0.0425,10,28.13,1306159,447228,436,162,0,3.573253,0.0,119.75,200,5,1,34.37499999999999,182.05882352941174,28.915406828605395,1 +20,207,288,4,292,7447,593,1846,4841,168,0,0.0556,1,27.95,1306154,448433,0,0,0,0.0,0.0,118.875,200,5,1,5.817221223021583,209.27945143884895,5.6598963219025,1 +21,0,0,9,9,13711,506,574,12100,531,0,0.0309,10,28.25,1304825,447457,0,0,0,2.394262,0.0,123.125,200,5,1,0.0,693.3151294498382,0.0,1 +22,0,0,0,0,12244,127,732,11072,313,0,0.0487,10,28.46,1303741,446918,0,0,0,1.437008,0.0,123.75,200,5,1,0.0,392.8388090349076,0.0,1 +23,0,0,4,4,12866,287,1454,10596,529,0,0.0355,37,28.17,1304255,448422,0,0,0,0.0,0.0,122.0,200,5,1,0.0,566.2852112676056,0.0,1 +24,255,407,7,414,13906,435,1384,11186,901,0,0.042,37,28.08,1305075,448422,0,0,0,0.296055,0.0,120.625,200,5,1,9.486607142857142,517.3363095238095,9.315779883381925,1 +25,802,1373,0,1373,12798,556,1636,9835,771,0,0.0649,3,27.86,1305333,449713,0,0,53,3.405214,0.0,120.875,200,5,1,19.30855161787365,308.11825885978425,18.1699149709961,1 +26,114,179,9,188,7517,429,789,5811,488,0,0.0466,10,28.04,1303973,449577,0,0,0,5.142853,0.0,121.875,200,5,1,3.8224248927038627,252.04533261802572,3.765321441286193,1 +27,857,1191,62,1253,8051,631,907,6189,324,0,0.0411,6,27.9,1303861,450595,242,0,0,0.372647,0.0,122.25,200,5,1,32.58059610705596,306.07512165450123,29.44615842589892,1 +28,86,112,53,165,19448,705,2055,16023,665,0,0.0596,10,28.22,1302243,449723,0,0,563,3.232071,0.0,122.125,200,5,1,2.2546140939597317,509.8573825503356,2.244687974778789,1 +29,144,200,40,240,10131,241,730,8711,448,0,0.0359,10,28.23,1303147,448956,0,0,0,0.1007179999999999,0.0,122.25,200,5,1,6.2674094707520895,440.93837047353765,6.17957424313279,1 +30,0,0,0,0,6615,853,2246,3321,195,0,0.0355,10,28.32,1303381,448113,0,0,0,2.547351,0.0,125.75,200,5,1,0.0,291.15316901408454,0.0,1 +31,0,0,0,0,2888,2,2,2883,1,0,0.0264,10,28.53,1302987,447039,0,0,0,2.399584,0.0,124.125,200,5,1,0.0,170.92803030303028,0.0,1 +32,1,5,0,5,1634,21,29,1376,208,0,0.0886,10,28.47,1301930,448259,0,0,0,36.879346000000005,0.0,124.0,200,5,1,0.01763544018058691,28.816309255079005,0.017624653978641595,1 +33,0,0,16,16,8908,249,321,7738,600,0,0.0783,10,28.79,1300543,447235,0,0,357,4.631443,0.0,122.875,200,5,1,0.0,177.76181353767564,0.0,1 +34,0,0,0,0,4455,1,4,3894,556,0,0.0433,10,28.96,1299479,446952,0,0,0,6.678808999999999,0.0,122.25,200,5,1,0.0,160.76068129330255,0.0,1 +35,676,833,1831,2664,1476,150,523,713,90,0,0.0456,1,28.76,1299451,448338,0,0,4916,0.992433,0.0,121.5,200,5,1,23.163377192982455,50.57565789473684,15.887149041283507,1 +36,0,0,78,78,13823,180,375,12538,730,0,0.0507,10,28.55,1300642,448757,0,358,0,0.8345889999999999,0.0,123.375,200,5,1,0.0,426.0046844181459,0.0,1 +37,0,0,0,0,21196,759,2724,15554,2159,0,0.0505,10,28.43,1300499,449711,0,0,0,0.0,0.0,121.375,200,5,1,0.0,655.8168316831683,0.0,1 +38,0,0,0,0,21170,763,2013,17304,1090,0,0.0544,10,28.33,1300211,450601,0,0,0,0.0,0.0,119.25,200,5,1,0.0,608.053768382353,0.0,1 +39,29,53,0,53,16447,735,1546,13192,974,0,0.0428,15,28.17,1301423,450694,0,0,271,0.0,0.0,122.375,200,5,1,1.0587032710280373,600.4307827102804,1.0568398093346765,1 +40,525,702,0,702,21881,936,2303,17050,1592,0,0.0671,5,28.03,1302525,450809,0,0,350,0.459977,0.0,122.0,200,5,1,12.225223546944857,509.5240312965723,11.938771598263875,1 +41,2726,3757,0,3757,1927,147,982,712,86,0,0.0557,5,27.83,1302670,452054,0,0,0,0.986873,0.0,119.5,200,5,1,76.46992818671455,54.05632854578098,31.66936419853835,1 +42,2234,3018,77,3095,4397,80,478,3536,303,0,0.0638,8,27.9,1301162,452771,0,57,0,1.724588,0.0,114.5,200,5,1,54.7119905956113,107.68514890282134,36.27938812379775,1 +43,155,193,24,217,8356,355,944,6610,447,0,0.0535,10,28.07,1301005,451768,0,0,0,0.410625,0.0,117.125,200,5,1,4.526869158878505,244.04205607476635,4.444427058111712,1 +44,749,957,0,957,8498,486,1105,6536,371,0,0.0443,8,28.23,1299805,451689,0,0,0,0.268704,0.0,115.625,200,5,1,26.417889390519186,299.7319413092551,24.278060348289397,1 +45,642,921,309,1230,1883,65,323,1231,264,0,0.0313,8,28.16,1299436,452448,0,0,0,2.502489,0.0,111.875,200,5,1,32.048722044728436,93.99960063897764,23.900096479296494,1 +46,213,368,9,377,2037,297,565,1076,99,0,0.036,8,28.04,1299201,453486,0,0,0,0.199429,0.0,109.625,200,5,1,9.244791666666668,88.41145833333334,8.369618055555556,1 +47,1661,2464,113,2577,2936,231,1166,1397,143,0,0.0691,7,27.86,1299989,454066,0,0,0,0.083949,0.0,106.5,200,5,1,37.558791606367585,66.38929088277858,23.987951306568462,1 +49,1628,2203,243,2445,3492,327,952,2076,137,0,0.0619,9,28.31,1298621,452083,0,0,0,0.5209520000000001,0.0,112.375,200,5,1,41.094507269789986,88.14620355411955,28.027738161348957,1 +50,706,1157,39,1196,4650,523,790,3205,132,0,0.0756,12,28.45,1297617,452014,387,0,0,23.046204,0.0,113.875,200,5,1,14.591600529100528,96.1061507936508,12.668211811112295,1 +53,683,1012,205,1217,154,10,43,68,33,0,0.0381,5,28.92,1296655,449590,0,0,0,2.709364,0.0,113.0,200,5,1,28.010170603674542,6.315616797900263,5.153603671404873,1 +54,1737,2669,99,2767,8325,157,728,6907,534,0,0.0632,7,28.71,1297131,450617,0,0,0,5.440506,0.0,115.375,200,5,1,42.94402689873417,205.81981803797464,35.53061259510653,1 +55,710,1038,136,1174,18900,1233,1294,15302,1071,0,0.0552,6,28.5,1298839,450627,0,0,0,0.054911,0.0,117.75,200,5,1,20.097373188405797,534.9864130434783,19.369727346296255,1 +56,467,830,1370,2200,9812,563,1487,7359,403,0,0.0652,2,28.66,1298820,449535,0,0,5479,3.030183,0.0,118.875,200,5,1,11.191526073619633,235.1418711656442,10.68306779203773,1 +57,1092,1535,1679,3214,1847,40,266,1405,136,0,0.0531,5,28.85,1297625,449235,0,0,3088,0.3413199999999999,0.0,118.25,200,5,1,32.13276836158192,54.34910546139359,20.193679198312967,1 +58,320,442,1948,2390,724,43,179,471,31,0,0.0539,2,28.87,1298550,448272,0,592,7444,1.075076,0.0,118.5,200,5,1,9.276437847866418,20.987940630797773,6.433085250819241,1 +59,752,1131,0,1131,4750,26,69,3873,783,0,0.0881,5,29.15,1297439,447302,0,0,0,10.990833,0.0,113.5,200,5,1,13.337116912599319,84.24375709421112,11.51423215827822,1 +60,489,688,8,696,2981,301,500,2038,143,0,0.0743,17,29.12,1296503,448319,0,0,0,31.479772,0.0,114.75,200,5,1,10.283479138627186,62.68926648721399,8.834308735518052,1 diff --git a/activitysim/examples/prototype_mwcog/test/regress/final_persons.csv b/activitysim/examples/prototype_mwcog/test/regress/final_persons.csv new file mode 100644 index 0000000000..d6328451a4 --- /dev/null +++ b/activitysim/examples/prototype_mwcog/test/regress/final_persons.csv @@ -0,0 +1,24 @@ +person_id,puma_geoid,TAZ,household_id,per_num,age,SEX,WKHP,WKW,ESR,RAC1P,HISP,SCHG,MIL,NAICSP,INDP,age_0_to_5,age_6_to_12,age_16_to_19,age_16_p,adult,young,old,male,female,esr,wkhp,wkw,schg,mil,pemploy,pstudent,ptype,has_non_worker,has_retiree,has_preschool_kid,has_driving_kid,has_school_kid,has_full_time,has_part_time,has_university,student_is_employed,nonstudent_to_school,is_student,is_gradeschool,is_highschool,is_university,school_segment,is_worker,home_zone_id,PNUM,income,income_in_thousands,income_segment,is_fulltime_worker,is_parttime_worker,value_of_time,is_income_less25K,is_income_25K_to_60K,is_income_60K_to_120K,is_income_greater60K,is_income_greater120K,is_non_worker_in_HH,is_all_adults_full_time_workers,is_pre_drive_child_in_HH,has_young_children,has_children_6_to_12,hh_child,school_zone_id,distance_to_school,roundtrip_auto_time_to_school,workplace_zone_id,workplace_location_logsum,distance_to_work,workplace_in_cbd,work_zone_area_type,roundtrip_auto_time_to_work,work_auto_savings,work_auto_savings_ratio,work_from_home,transit_pass_subsidy,transit_pass_ownership,free_parking_at_work,telecommute_frequency,cdap_activity,travel_active,under16_not_at_school,has_preschool_kid_at_home,has_school_kid_at_home,mandatory_tour_frequency,work_and_school_and_worker,work_and_school_and_student,num_mand,num_work_tours,has_pre_school_child_with_mandatory,has_driving_age_child_with_mandatory,num_joint_tours,non_mandatory_tour_frequency,num_non_mand,num_escort_tours,num_eatout_tours,num_shop_tours,num_maint_tours,num_discr_tours,num_social_tours,num_non_escort_tours,num_shop_maint_tours,num_shop_maint_escort_tours,num_add_shop_maint_tours,num_soc_discr_tours,num_add_soc_discr_tours +159001,1100105,25,1590,1,34,1,4,6,2,2,1,-9,4,711M,8563.0,False,False,False,True,True,False,False,True,False,2,4,6,-9,4,2,3.0,2,False,False,True,False,True,True,False,False,False,False,False,False,False,False,0,True,25,1,17192,17.192,1,False,True,2.1180144126507763,True,False,False,False,False,False,False,True,True,True,7,-1,,0.0,27,17.26930561846995,0.5,True,1.0,2.3200002,15.68,0.13066667,False,1,1,False,No_Telecommute,H,False,False,True,True,,False,False,0,0,True,False,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 +159002,1100105,25,1590,2,9,1,-9,-9,-9,2,1,6,-9,0,0.0,False,True,False,False,False,True,False,True,False,-9,-9,-9,6,-9,4,1.0,7,False,False,True,False,True,True,True,False,False,False,True,True,False,False,1,False,25,2,17192,17.192,1,False,False,1.4127156132380678,True,False,False,False,False,False,False,True,True,True,7,19,0.6,3.04,-1,,,False,,0.0,0.0,0.0,False,1,1,False,,H,False,True,True,True,,False,False,0,0,True,False,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 +159003,1100105,25,1590,3,7,1,-9,-9,-9,2,1,3,-9,0,0.0,False,True,False,False,False,True,False,True,False,-9,-9,-9,3,-9,4,1.0,7,False,False,True,False,True,True,True,False,False,False,True,True,False,False,1,False,25,3,17192,17.192,1,False,False,1.4127156132380678,True,False,False,False,False,False,False,True,True,True,7,27,0.5,2.3200002,-1,,,False,,0.0,0.0,0.0,False,1,0,False,,M,True,False,True,True,school1,False,False,1,0,True,False,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 +159004,1100105,25,1590,4,3,2,-9,-9,-9,2,1,1,-9,0,0.0,True,False,False,False,False,True,False,False,True,-9,-9,-9,1,-9,4,1.0,8,False,False,True,False,True,True,True,False,False,False,True,True,False,False,1,False,25,4,17192,17.192,1,False,False,1.4127156132380678,True,False,False,False,False,False,False,True,True,True,7,27,0.5,2.3200002,-1,,,False,,0.0,0.0,0.0,False,1,0,False,,H,False,True,True,True,,False,False,0,0,True,False,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 +159005,1100105,25,1590,5,2,2,-9,-9,-9,2,1,-9,-9,0,0.0,True,False,False,False,False,True,False,False,True,-9,-9,-9,-9,-9,4,1.0,8,False,False,True,False,True,True,True,False,False,False,True,True,False,False,1,False,25,5,17192,17.192,1,False,False,1.4127156132380678,True,False,False,False,False,False,False,True,True,True,7,27,0.5,2.3200002,-1,,,False,,0.0,0.0,0.0,False,1,0,False,,M,True,False,True,True,school1,False,False,1,0,True,False,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 +159006,1100105,25,1590,6,1,1,-9,-9,-9,2,1,-9,-9,0,0.0,True,False,False,False,False,True,False,True,False,-9,-9,-9,-9,-9,4,1.0,8,False,False,True,False,True,True,True,False,False,False,True,True,False,False,1,False,25,6,17192,17.192,1,False,False,1.4127156132380678,True,False,False,False,False,False,False,True,True,True,7,19,0.6,3.04,-1,,,False,,0.0,0.0,0.0,False,1,0,False,,H,False,True,True,True,,False,False,0,0,True,False,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 +159007,1100105,25,1590,7,0,2,-9,-9,-9,2,1,-9,-9,0,0.0,True,False,False,False,False,True,False,False,True,-9,-9,-9,-9,-9,4,1.0,8,False,False,True,False,True,True,True,False,False,False,True,True,False,False,1,False,25,7,17192,17.192,1,False,False,1.4127156132380678,True,False,False,False,False,False,False,True,True,True,7,19,0.6,3.04,-1,,,False,,0.0,0.0,0.0,False,1,0,False,,M,True,False,True,True,school1,False,False,1,0,True,False,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 +159008,1100105,25,1590,8,7,2,-9,-9,-9,2,1,-9,-9,0,0.0,False,True,False,False,False,True,False,False,True,-9,-9,-9,-9,-9,4,1.0,7,False,False,True,False,True,True,True,False,False,False,True,True,False,False,1,False,25,8,17192,17.192,1,False,False,1.4127156132380678,True,False,False,False,False,False,False,True,True,True,7,27,0.5,2.3200002,-1,,,False,,0.0,0.0,0.0,False,1,1,False,,H,False,True,True,True,,False,False,0,0,True,False,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 +159009,1100105,25,1590,9,26,2,40,3,1,2,1,15,4,4533,5490.0,False,False,False,True,True,False,False,False,True,1,40,3,15,4,1,3.0,1,False,False,True,False,True,False,True,False,False,False,False,False,False,False,0,True,25,9,17192,17.192,1,True,False,2.1180144126507763,True,False,False,False,False,False,False,True,True,True,7,-1,,0.0,50,17.280045991201952,2.0,True,1.0,8.719999,69.28,0.57733333,True,1,0,False,No_Telecommute,H,False,False,True,True,,False,False,0,0,True,False,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 +162001,1100105,25,1620,1,31,1,40,3,3,1,1,-9,4,8139Z,9190.0,False,False,False,True,True,False,False,True,False,3,40,3,-9,4,3,3.0,4,False,False,False,False,False,True,False,False,False,False,False,False,False,False,0,False,25,1,168841,168.841,4,False,False,8.070805737817167,False,False,False,True,True,True,False,False,False,False,0,-1,,0.0,-1,,,False,,0.0,0.0,0.0,False,1,0,False,,N,True,False,False,False,,False,False,0,0,False,False,0,17,2,0,0,1,0,1,0,2,1,1,0,1,0 +162002,1100105,25,1620,2,34,1,40,1,1,1,1,-9,4,5121,6570.0,False,False,False,True,True,False,False,True,False,1,40,1,-9,4,1,3.0,1,True,False,False,False,False,True,False,False,False,False,False,False,False,False,0,True,25,2,168841,168.841,4,True,False,8.070805737817167,False,False,False,True,True,True,False,False,False,False,0,-1,,0.0,38,15.56892502432064,1.1,True,1.0,5.68,38.32,0.31933334,False,1,0,False,No_Telecommute,M,True,False,False,False,work1,False,False,1,1,False,False,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 +162003,1100105,25,1620,3,27,1,45,1,1,1,1,-9,4,5417,7460.0,False,False,False,True,True,False,False,True,False,1,45,1,-9,4,1,3.0,1,True,False,False,False,False,True,False,False,False,False,False,False,False,False,0,True,25,3,168841,168.841,4,True,False,8.070805737817167,False,False,False,True,True,True,False,False,False,False,0,-1,,0.0,26,15.591843366919742,0.3,True,1.0,1.6600001,10.34,0.086166665,False,1,1,False,1_day_week,M,True,False,False,False,work1,False,False,1,1,False,False,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 +163201,1100105,25,1632,1,34,1,60,1,1,1,1,-9,4,92113,9380.0,False,False,False,True,True,False,False,True,False,1,60,1,-9,4,1,3.0,1,False,False,True,False,False,False,False,True,False,False,False,False,False,False,0,True,25,1,107067,107.067,3,True,False,9.177177574953367,False,False,True,True,False,False,False,False,True,False,1,-1,,0.0,24,15.60895053515653,0.3,True,1.0,1.78,10.22,0.08516667,False,1,0,True,No_Telecommute,M,True,False,False,False,work1,False,False,1,1,False,False,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 +163202,1100105,25,1632,2,33,2,-9,-9,6,1,1,16,4,5413,7290.0,False,False,False,True,True,False,False,False,True,6,-9,-9,16,4,3,2.0,3,False,False,True,False,False,True,False,False,False,False,True,False,False,True,3,False,25,2,107067,107.067,3,False,False,9.177177574953367,False,False,True,True,False,False,False,False,True,False,1,35,1.3,6.51,-1,,,False,,0.0,0.0,0.0,False,1,0,False,,N,True,False,False,False,,False,False,0,0,False,False,0,64,2,2,0,0,0,0,0,0,0,2,0,0,0 +163203,1100105,25,1632,3,3,1,-9,-9,-9,1,1,1,-9,0,0.0,True,False,False,False,False,True,False,True,False,-9,-9,-9,1,-9,4,1.0,8,False,False,False,False,False,True,False,True,False,False,True,True,False,False,1,False,25,3,107067,107.067,3,False,False,6.121177442493896,False,False,True,True,False,False,False,False,True,False,1,19,0.6,3.04,-1,,,False,,0.0,0.0,0.0,False,1,0,False,,N,True,True,False,False,,False,False,0,0,False,False,0,16,1,0,0,1,0,0,0,1,1,1,0,0,0 +848401,1100105,42,8484,1,29,1,60,1,1,1,1,-9,4,5412,7280.0,False,False,False,True,True,False,False,True,False,1,60,1,-9,4,1,3.0,1,False,False,False,False,False,False,False,False,False,False,False,False,False,False,0,True,42,1,175104,175.104,4,True,False,14.589903621439268,False,False,False,True,True,False,True,False,False,False,0,-1,,0.0,1,16.987266532171322,1.6,True,1.0,7.86,60.14,0.50116664,False,1,0,True,No_Telecommute,M,True,False,False,False,work1,False,False,1,1,False,False,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 +1047301,1100105,44,10473,1,25,2,40,1,1,6,1,-9,4,611M1,7870.0,False,False,False,True,True,True,False,False,True,1,40,1,-9,4,1,3.0,1,False,False,False,False,False,False,False,False,False,False,False,False,False,False,0,True,44,1,47109,47.109,1,True,False,4.307904952573323,False,True,False,False,False,False,True,False,False,False,0,-1,,0.0,54,15.952344283146994,0.6,True,1.0,3.15,20.85,0.17375,False,1,0,True,No_Telecommute,H,False,False,False,False,,False,False,0,0,False,False,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 +1163601,1100105,47,11636,1,44,1,40,1,4,1,19,16,1,928110P3,9690.0,False,False,False,True,True,False,False,True,False,4,40,1,16,1,1,3.0,1,False,False,False,False,False,True,False,False,False,False,False,False,False,False,0,True,47,1,265310,265.31,4,True,False,10.475855530295568,False,False,False,True,True,False,True,False,False,False,0,-1,,0.0,55,16.06131219150949,1.1,True,1.0,5.0299997,34.97,0.29141667,False,1,0,False,No_Telecommute,M,True,False,False,False,work1,False,False,1,1,False,False,1,0,0,0,0,1,0,0,0,0,1,1,0,0,0 +1163602,1100105,47,11636,2,27,2,50,1,1,1,1,-9,4,8139Z,9190.0,False,False,False,True,True,False,False,False,True,1,50,1,-9,4,1,3.0,1,False,False,False,False,False,True,False,False,False,False,False,False,False,False,0,True,47,2,265310,265.31,4,True,False,10.475855530295568,False,False,False,True,True,False,True,False,False,False,0,-1,,0.0,40,15.99097616471269,1.1,True,1.0,5.1099997,38.89,0.32408333,False,1,1,True,No_Telecommute,M,True,False,False,False,work1,False,False,1,1,False,False,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 +1268201,1100105,47,12682,1,32,1,45,1,1,9,1,-9,4,813M,9170.0,False,False,False,True,True,False,False,True,False,1,45,1,-9,4,1,3.0,1,False,False,False,False,False,False,False,False,False,False,False,False,False,False,0,True,47,1,62099,62.099,2,True,False,21.56089619468371,False,False,True,True,False,False,True,False,False,False,0,-1,,0.0,12,16.770280305772175,2.8,True,1.0,11.34,65.61,0.54675,False,1,1,True,No_Telecommute,M,True,False,False,False,work1,False,False,1,1,False,False,0,4,1,0,1,0,0,0,0,1,0,0,0,0,0 +1428101,1100105,49,14281,1,39,1,50,1,1,1,2,-9,4,481,6070.0,False,False,False,True,True,False,False,True,False,1,50,1,-9,4,1,3.0,1,False,False,False,False,False,False,False,False,False,False,False,False,False,False,0,True,49,1,73804,73.804,2,True,False,9.579868420839766,False,False,True,True,False,False,True,False,False,False,0,-1,,0.0,21,15.591595223780121,2.0,True,1.0,9.05,60.63,0.50525004,True,1,1,False,No_Telecommute,N,True,False,False,False,,False,False,0,0,False,False,0,1,1,0,0,0,0,1,0,1,0,0,0,1,0 +1984001,1100105,57,19840,1,27,1,40,1,1,1,1,-9,4,928P,9590.0,False,False,False,True,True,False,False,True,False,1,40,1,-9,4,1,3.0,1,False,False,False,False,False,False,False,False,False,False,False,False,False,False,0,True,57,1,95289,95.289,2,True,False,16.892032842005793,False,False,True,True,False,False,True,False,False,False,0,-1,,0.0,28,16.95193347497167,1.0,True,1.0,4.7200003,35.28,0.294,False,1,1,False,No_Telecommute,M,True,False,False,False,work1,False,False,1,1,False,False,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 +270473601,1100105,36,2704736,1,47,1,40,3,1,2,1,-9,4,722Z,8680.0,False,False,False,True,True,False,False,True,False,1,40,3,-9,4,1,3.0,1,False,False,False,False,False,False,False,False,False,False,False,False,False,False,0,True,36,1,20021,20.021,1,True,False,5.707035026149024,True,False,False,False,False,False,True,False,False,False,0,-1,,0.0,37,16.11584096402891,0.2,True,1.0,1.96,14.04,0.117,False,1,1,False,No_Telecommute,H,False,False,False,False,,False,False,0,0,False,False,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 diff --git a/activitysim/examples/prototype_mwcog/test/regress/final_tours.csv b/activitysim/examples/prototype_mwcog/test/regress/final_tours.csv new file mode 100644 index 0000000000..057e169fee --- /dev/null +++ b/activitysim/examples/prototype_mwcog/test/regress/final_tours.csv @@ -0,0 +1,22 @@ +tour_id,person_id,tour_type,tour_type_count,tour_type_num,tour_num,tour_count,tour_category,number_of_participants,destination,origin,household_id,tdd,start,end,duration,composition,destination_logsum,tour_mode,mode_choice_logsum,atwork_subtour_frequency,parent_tour_id,stop_frequency,primary_purpose +6519154,159003,school,1,1,1,1,mandatory,1,27.0,25.0,1590,451.0,11.0,27.0,16.0,,,WALK,2.758076567942611,,,0out_0in,school +6519236,159005,school,1,1,1,1,mandatory,1,27.0,25.0,1590,293.0,7.0,27.0,20.0,,,WALK,2.7353689305227307,,,0out_0in,school +6519318,159007,school,1,1,1,1,mandatory,1,19.0,25.0,1590,451.0,11.0,27.0,16.0,,,WALK,2.713144660446043,,,0out_0in,school +6642121,162002,work,1,1,1,1,mandatory,1,38.0,25.0,1620,604.0,15.0,38.0,23.0,,,KNR_MR,2.7209542315290243,eat,,0out_0in,work +6642162,162003,work,1,1,1,1,mandatory,1,26.0,25.0,1620,527.0,13.0,30.0,17.0,,,WALK,3.5666322634386933,eat,,0out_0in,work +6691280,163201,work,1,1,1,1,mandatory,1,24.0,25.0,1632,380.0,9.0,33.0,24.0,,,PNR_MR,1.1380701762124639,no_subtours,,0out_0in,work +34784480,848401,work,1,1,1,1,mandatory,1,1.0,42.0,8484,629.0,16.0,30.0,14.0,,,DRIVEALONE,0.34930891955963855,no_subtours,,0out_0in,work +47707680,1163601,work,1,1,1,1,mandatory,1,55.0,47.0,11636,570.0,14.0,38.0,24.0,,,DRIVEALONE,0.3246213622382585,no_subtours,,0out_2in,work +47707721,1163602,work,1,1,1,1,mandatory,1,40.0,47.0,11636,448.0,11.0,24.0,13.0,,,PNR_AB,0.529349080222066,no_subtours,,0out_0in,work +51996280,1268201,work,1,1,1,1,mandatory,1,12.0,47.0,12682,239.0,6.0,15.0,9.0,,,DRIVEALONE,-0.21758026864256086,no_subtours,,0out_0in,work +81344080,1984001,work,1,1,1,1,mandatory,1,28.0,57.0,19840,360.0,9.0,13.0,4.0,,,WALK_MR,0.516684431149219,no_subtours,,0out_0in,work +47707660,1163601,shopping,1,1,1,1,joint,2,25.0,47.0,11636,1114.0,38.0,42.0,4.0,adults,14.401517047688447,SHARED2,-0.0919298874764291,,,0out_1in,shopping +6642074,162001,shopping,1,1,1,2,non_mandatory,1,24.0,25.0,1620,742.0,20.0,21.0,1.0,,15.04014514701597,WALK,1.7064715572448472,,,0out_0in,shopping +6642066,162001,othdiscr,1,1,2,2,non_mandatory,1,25.0,25.0,1620,516.0,13.0,19.0,6.0,,15.116038427975054,WALK,2.012854024573718,,,0out_0in,othdiscr +6691291,163202,escort,2,1,1,2,non_mandatory,1,39.0,25.0,1632,946.0,28.0,29.0,1.0,,13.51585783844801,SHARED2,0.021552688601505547,,,0out_0in,escort +6691292,163202,escort,2,2,2,2,non_mandatory,1,23.0,25.0,1632,852.0,24.0,25.0,1.0,,13.533137735669488,WALK,0.8533704840550164,,,0out_0in,escort +6691356,163203,shopping,1,1,1,1,non_mandatory,1,22.0,25.0,1632,690.0,18.0,28.0,10.0,,14.593476482584549,WALK,0.041425542945739295,,,0out_0in,shopping +51996247,1268201,eatout,1,1,1,1,non_mandatory,1,39.0,47.0,12682,713.0,19.0,21.0,2.0,,13.057026256478705,WALK,0.3187482301360538,,,0out_0in,eatout +58552166,1428101,othdiscr,1,1,1,1,non_mandatory,1,38.0,49.0,14281,879.0,25.0,28.0,3.0,,14.354797920912523,WALK,0.6250888340229916,,,1out_3in,othdiscr +6642086,162002,eat,1,1,1,1,atwork,1,38.0,38.0,1620,799.0,22.0,23.0,1.0,,15.4963111559122,WALK,2.9189754365251126,,6642121.0,0out_0in,atwork +6642127,162003,eat,1,1,1,1,atwork,1,40.0,26.0,1620,711.0,19.0,19.0,0.0,,15.418258993054184,WALK,1.8680736528431738,,6642162.0,0out_0in,atwork diff --git a/activitysim/examples/prototype_mwcog/test/regress/final_trips.csv b/activitysim/examples/prototype_mwcog/test/regress/final_trips.csv index 81d05d0dd2..b69e6a62d8 100644 --- a/activitysim/examples/prototype_mwcog/test/regress/final_trips.csv +++ b/activitysim/examples/prototype_mwcog/test/regress/final_trips.csv @@ -6,9 +6,9 @@ trip_id,person_id,household_id,primary_purpose,trip_num,outbound,trip_count,dest 52154545,159007,1590,school,1,True,1,19,25,6519318,school,,11,WALK,3.1673109534958903 52154549,159007,1590,school,1,False,1,25,19,6519318,home,,27,WALK,3.1673109534958903 53136529,162001,1620,othdiscr,1,True,1,25,25,6642066,othdiscr,,13,WALK,2.0354872001527298 -53136533,162001,1620,othdiscr,1,False,1,25,25,6642066,home,,17,WALK,2.0354872001527298 +53136533,162001,1620,othdiscr,1,False,1,25,25,6642066,home,,19,WALK,2.0354872001527298 53136593,162001,1620,shopping,1,True,1,24,25,6642074,shopping,,20,WALK,0.5873669693786134 -53136597,162001,1620,shopping,1,False,1,25,24,6642074,home,,23,WALK,0.5873669693786134 +53136597,162001,1620,shopping,1,False,1,25,24,6642074,home,,21,WALK,0.5873669693786134 53136689,162002,1620,atwork,1,True,1,38,38,6642086,atwork,,22,WALK,-0.611999989181757 53136693,162002,1620,atwork,1,False,1,38,38,6642086,work,,23,WALK,-0.611999989181757 53136969,162002,1620,work,1,True,1,38,25,6642121,work,,15,KNR_MR,-0.8976646759623489 @@ -17,19 +17,19 @@ trip_id,person_id,household_id,primary_purpose,trip_num,outbound,trip_count,dest 53137021,162003,1620,atwork,1,False,1,26,40,6642127,work,,19,WALK,0.8548156977447776 53137297,162003,1620,work,1,True,1,26,25,6642162,work,,13,WALK,-0.16346785221758944 53137301,162003,1620,work,1,False,1,25,26,6642162,home,,30,WALK,-0.16346785221758944 -53530241,163201,1632,work,1,True,1,24,25,6691280,work,,9,PNR_MR,-0.4392097253621265 +53530241,163201,1632,work,1,True,1,24,25,6691280,work,,9,PNR_MR,-0.4392097253621263 53530245,163201,1632,work,1,False,1,25,24,6691280,home,,33,PNR_MR,-0.2591827350249029 53530329,163202,1632,escort,1,True,1,39,25,6691291,escort,,28,DRIVEALONE,-0.06197964360005489 53530333,163202,1632,escort,1,False,1,25,39,6691291,home,,29,SHARED2,-0.1701462043767353 53530337,163202,1632,escort,1,True,1,23,25,6691292,escort,,24,WALK,-0.92999997921288 53530341,163202,1632,escort,1,False,1,25,23,6691292,home,,25,WALK,-0.92999997921288 -53530849,163203,1632,shopping,1,True,1,22,25,6691356,shopping,,19,WALK,-1.409999968484044 -53530853,163203,1632,shopping,1,False,1,25,22,6691356,home,,20,WALK,-1.289999971166253 +53530849,163203,1632,shopping,1,True,1,22,25,6691356,shopping,,18,WALK,-1.409999968484044 +53530853,163203,1632,shopping,1,False,1,25,22,6691356,home,,28,WALK,-1.289999971166253 278275841,848401,8484,work,1,True,1,1,42,34784480,work,,16,DRIVEALONE,-0.3047444691565592 278275845,848401,8484,work,1,False,1,42,1,34784480,home,,30,DRIVEALONE,-0.3737130471632631 -381661281,1163601,11636,shopping,1,True,1,25,47,47707660,shopping,,39,SHARED2,0.13405080458662966 -381661285,1163601,11636,shopping,1,False,2,37,25,47707660,shopping,12.938418648448124,40,SHARED2,-0.19334951027125485 -381661286,1163601,11636,shopping,2,False,2,47,37,47707660,home,,40,SHARED2,0.11558833086308477 +381661281,1163601,11636,shopping,1,True,1,25,47,47707660,shopping,,38,SHARED2,0.13405080458662966 +381661285,1163601,11636,shopping,1,False,2,37,25,47707660,shopping,12.616106774095424,41,SHARED2,-0.2823603086688386 +381661286,1163601,11636,shopping,2,False,2,47,37,47707660,home,,42,SHARED2,0.03716309792422327 381661441,1163601,11636,work,1,True,1,55,47,47707680,work,,14,DRIVEALONE,-0.2776450324248023 381661445,1163601,11636,work,1,False,3,49,55,47707680,shopping,8.755660261900594,37,DRIVEALONE,-1.318377909968898 381661446,1163601,11636,work,2,False,3,26,49,47707680,shopping,11.445671730580356,37,DRIVEALONE,-0.628015657476555 @@ -47,4 +47,4 @@ trip_id,person_id,household_id,primary_purpose,trip_num,outbound,trip_count,dest 468417335,1428101,14281,othdiscr,3,False,4,45,54,58552166,othmaint,26.28617562432176,26,WALK,4.039591798974065 468417336,1428101,14281,othdiscr,4,False,4,49,45,58552166,home,,28,WALK,2.895014407185858 650752641,1984001,19840,work,1,True,1,28,57,81344080,work,,9,WALK,0.21761578208744412 -650752645,1984001,19840,work,1,False,1,57,28,81344080,home,,13,WALK_MR,0.22380439703983218 +650752645,1984001,19840,work,1,False,1,57,28,81344080,home,,13,WALK_MR,0.22380439703983235 diff --git a/activitysim/examples/prototype_mwcog/test/test_mwcog.py b/activitysim/examples/prototype_mwcog/test/test_mwcog.py index bd23077689..98652504f6 100644 --- a/activitysim/examples/prototype_mwcog/test/test_mwcog.py +++ b/activitysim/examples/prototype_mwcog/test/test_mwcog.py @@ -2,6 +2,7 @@ # See full license in LICENSE.txt. import os import subprocess +import sys import pandas as pd import pandas.testing as pdt @@ -15,7 +16,7 @@ def teardown_function(func): inject.reinject_decorated_tables() -def test_mwcog(): +def _test_mwcog(sharrow=False): def example_path(dirname): resource = os.path.join("examples", "prototype_mwcog", dirname) return pkg_resources.resource_filename("activitysim", resource) @@ -34,12 +35,23 @@ def regress(): file_path = os.path.join(os.path.dirname(__file__), ".." + os.sep + "simulation.py") + if os.environ.get("GITHUB_ACTIONS") == "true": + executable = ["coverage", "run", "-a"] + else: + executable = [sys.executable] + + if sharrow: + sh_configs = ["-c", example_path("configs_sharrow")] + else: + sh_configs = [] + subprocess.run( - [ - "coverage", - "run", - "-a", + executable + + [ file_path, + ] + + sh_configs + + [ "-c", test_path("configs"), "-c", @@ -55,6 +67,14 @@ def regress(): regress() +def test_mwcog(): + _test_mwcog() + + +def test_mwcog_sharrow(): + _test_mwcog(sharrow=True) + + if __name__ == "__main__": test_mwcog() diff --git a/activitysim/examples/prototype_sandag_xborder/configs/legacy-1.0.4/logging.yaml b/activitysim/examples/prototype_sandag_xborder/configs/legacy-1.0.4/logging.yaml new file mode 100755 index 0000000000..1e92b76a32 --- /dev/null +++ b/activitysim/examples/prototype_sandag_xborder/configs/legacy-1.0.4/logging.yaml @@ -0,0 +1,53 @@ +# Config for logging +# ------------------ +# See http://docs.python.org/2.7/library/logging.config.html#configuration-dictionary-schema + +logging: + version: 1 + disable_existing_loggers: true + + + # Configuring the default (root) logger is highly recommended + root: + level: NOTSET + handlers: [console, logfile] + + loggers: + + activitysim: + level: DEBUG + handlers: [console, logfile] + propagate: false + + orca: + level: WARN + handlers: [console, logfile] + propagate: false + + handlers: + + logfile: + class: logging.FileHandler + filename: !!python/object/apply:activitysim.core.config.log_file_path ['activitysim.log'] + mode: w + formatter: fileFormatter + level: NOTSET + + console: + class: logging.StreamHandler + stream: ext://sys.stdout + formatter: simpleFormatter + level: NOTSET + + formatters: + + simpleFormatter: + class: logging.Formatter + # format: '%(levelname)s - %(name)s - %(message)s' + format: '%(levelname)s - %(message)s' + datefmt: '%d/%m/%Y %H:%M:%S' + + fileFormatter: + class: logging.Formatter + format: '%(asctime)s - %(levelname)s - %(name)s - %(message)s' + datefmt: '%d/%m/%Y %H:%M:%S' diff --git a/activitysim/examples/prototype_sandag_xborder/configs/legacy-1.0.4/settings.yaml b/activitysim/examples/prototype_sandag_xborder/configs/legacy-1.0.4/settings.yaml new file mode 100755 index 0000000000..66ce520127 --- /dev/null +++ b/activitysim/examples/prototype_sandag_xborder/configs/legacy-1.0.4/settings.yaml @@ -0,0 +1,157 @@ +inherit_settings: False + +# number of households to simulate + +households_sample_size: 0 +multiprocess: True +strict: False +mem_tick: 30 +num_processes: 30 +fail_fast: True + +############## +# +# chunking +# + +# chooser chunk size in gigabytes +# target top memory usage during activitysim run (including shared memory, loaded tables, and transient memory usage) +chunk_size: 400_000_000_000 +#chunk_size: 0 + +# minimum fraction of total chunk_size to reserve for adaptive chunking +min_available_chunk_ratio: 0.05 + +# initial number of chooser rows for first chunk in training mode +# when there is no pre-existing chunk_cache to set initial value +# ordinarily bigger is better as long as it is not so big it causes memory issues (e.g. accessibility with lots of zones) +default_initial_rows_per_chunk: 500 + +# method to calculate memory overhead when chunking is enabled +chunk_method: hybrid_uss + +# chunk training mode +# training to determine the chunking settings written to a cache file that is re-used for production runs +# training +# production +# disabled +chunk_training_mode: disabled + +# whether to preserve or delete subprocess chunk and mem logs when they are consolidated at end of multiprocess run +keep_chunk_logs: True +keep_mem_logs: True + +############## + +trace_hh_id: + +# input tables +input_table_list: + - tablename: households + filename: households_xborder.csv + index_col: household_id + + - tablename: persons + filename: persons_xborder.csv + index_col: person_id + + - tablename: land_use + filename: mazs_xborder.csv + index_col: zone_id + rename_columns: + MAZ: zone_id + + - tablename: tours + filename: tours_xborder.csv + index_col: tour_id + keep_columns: + - pass_type + - tour_type + - purpose_id + - tour_category + - number_of_participants + - tour_num + - tour_count + - household_id + - person_id + + +# set false to disable variability check in simple_simulate and interaction_simulate +check_for_variability: False + +# - shadow pricing global switches + +# turn shadow_pricing on and off for all models (e.g. school and work) +# shadow pricing is deprecated for less than full samples +# see shadow_pricing.yaml for additional settings +use_shadow_pricing: False + +# turn writing of sample_tables on and off for all models +# (if True, tables will be written if DEST_CHOICE_SAMPLE_TABLE_NAME is specified in individual model settings) +want_dest_choice_sample_tables: False +want_dest_choice_presampling: True + +#resume_after: trip_scheduling + +models: + - initialize_landuse + - initialize_households + - initialize_tours + # --- STATIC cache prebuild steps + # single-process step to create attribute_combination list + - initialize_los + # multi-processable step to build STATIC cache + # (this step is a NOP if cache already exists and network_los.rebuild_tvpb_cache setting is False) + - initialize_tvpb + # # --- + - tour_scheduling_probabilistic + - tour_od_choice + - reassign_tour_purpose_by_poe + - tour_mode_choice_simulate + - stop_frequency + - trip_purpose + - trip_scheduling + - trip_destination + - trip_mode_choice + - write_trip_matrices + - write_tables + +multiprocess_steps: + - name: mp_initialize + begin: initialize_landuse + - name: mp_tvpb + begin: initialize_tvpb + num_processes: 5 + slice: + tables: + - attribute_combinations + - name: mp_households + begin: tour_scheduling_probabilistic + slice: + tables: + - households + - persons + - name: mp_summarize + begin: write_trip_matrices + +output_tables: + h5_store: False + action: include + prefix: final_ + # FIXME sort is an undocumented feature - sorts table by best index or ref_col according to traceable_table_indexes + sort: True + tables: + - checkpoints + - accessibility + - land_use + - households + - persons + - tours + - trips + - attribute_combinations + +output_summaries: + tours: + - tour_mode + - od_path_set + - do_path_set diff --git a/activitysim/examples/prototype_sandag_xborder/configs/legacy-1.0.4/trip_destination_annotate_trips_preprocessor.csv b/activitysim/examples/prototype_sandag_xborder/configs/legacy-1.0.4/trip_destination_annotate_trips_preprocessor.csv new file mode 100644 index 0000000000..d7563cc825 --- /dev/null +++ b/activitysim/examples/prototype_sandag_xborder/configs/legacy-1.0.4/trip_destination_annotate_trips_preprocessor.csv @@ -0,0 +1,7 @@ +Description,Target,Expression +#,, +,tour_mode,"reindex(tours.tour_mode, df.tour_id)" +,tour_duration,"reindex(((tours.end - tours.start + 1) * 0.5), df.tour_id)" +,tour_leg_dest,"np.where(df.outbound,reindex(tours.destination, df.tour_id), reindex(tours.origin, df.tour_id))" +,_tod,"np.where(df.outbound,reindex_i(tours.start, df.tour_id), reindex_i(tours.end, df.tour_id))" +,trip_period,network_los.skim_time_period_label(_tod) diff --git a/activitysim/examples/prototype_sandag_xborder/configs/logging.yaml b/activitysim/examples/prototype_sandag_xborder/configs/logging.yaml index df20cf0c7e..33072a571b 100755 --- a/activitysim/examples/prototype_sandag_xborder/configs/logging.yaml +++ b/activitysim/examples/prototype_sandag_xborder/configs/logging.yaml @@ -24,6 +24,18 @@ logging: handlers: [console, logfile] propagate: false + filelock: + level: WARN + + sharrow: + level: INFO + + blib2to3: + level: WARN + + black: + level: WARN + handlers: logfile: @@ -36,7 +48,7 @@ logging: console: class: logging.StreamHandler stream: ext://sys.stdout - formatter: simpleFormatter + formatter: elapsedFormatter level: NOTSET formatters: @@ -52,3 +64,7 @@ logging: format: '%(asctime)s - %(levelname)s - %(name)s - %(message)s' datefmt: '%d/%m/%Y %H:%M:%S' + elapsedFormatter: + (): activitysim.core.tracing.ElapsedTimeFormatter + format: '[{elapsedTime}] {levelname:s}: {message:s}' + style: '{' diff --git a/activitysim/examples/prototype_sandag_xborder/configs/settings.yaml b/activitysim/examples/prototype_sandag_xborder/configs/settings.yaml index 66ce520127..33bb5b8401 100755 --- a/activitysim/examples/prototype_sandag_xborder/configs/settings.yaml +++ b/activitysim/examples/prototype_sandag_xborder/configs/settings.yaml @@ -60,6 +60,15 @@ input_table_list: index_col: zone_id rename_columns: MAZ: zone_id + recode_columns: + zone_id: zero-based + TAZ: land_use_taz.TAZ + + - tablename: land_use_taz + filename: taz_xborder.csv + index_col: TAZ + recode_columns: + TAZ: zero-based - tablename: tours filename: tours_xborder.csv @@ -147,7 +156,10 @@ output_tables: - households - persons - tours - - trips + - tablename: trips + decode_columns: + origin: land_use.zone_id + destination: land_use.zone_id - attribute_combinations output_summaries: diff --git a/activitysim/examples/prototype_sandag_xborder/configs/trip_destination.csv b/activitysim/examples/prototype_sandag_xborder/configs/trip_destination.csv index 12fe38ccd9..c1dd27a274 100644 --- a/activitysim/examples/prototype_sandag_xborder/configs/trip_destination.csv +++ b/activitysim/examples/prototype_sandag_xborder/configs/trip_destination.csv @@ -1,6 +1,6 @@ Description,Expression,work,school,visit,shop,other -size term for purpose,"@np.log1p(size_terms.get(df.trip_dest, df.purpose))",1,1,1,1,1 -no attractions for purpose,"@size_terms.get(df.trip_dest, df.purpose) == 0",-999,-999,-999,-999,-999 +Size variable,"@np.log1p(size_terms.get(df.trip_dest, df.purpose)) # sharrow: np.log1p(size_terms['arry'])",1,1,1,1,1 +No attractions,"@size_terms.get(df.trip_dest, df.purpose) == 0 # sharrow: size_terms['arry'] == 0",-999,-999,-999,-999,-999 ,"_od_DIST@odt_skims['SOV_NT_M_DIST']",1,1,1,1,1 ,"_dp_DIST@dpt_skims['SOV_NT_M_DIST']",1,1,1,1,1 intermediate stops on half-tour,_stops_on_leg@df.trip_count - 1,1,1,1,1,1 @@ -13,4 +13,4 @@ last outbound intermediate stop,"@(df.trip_num == _stops_on_leg) & df.outbound & last outbound intermediate stop,"@(df.trip_num == _stops_on_leg) & df.outbound * np.clip(_dp_DIST - 1, 0, 3)",coef_last_outbound_dist_dest_gt1_max3,coef_last_outbound_dist_dest_gt1_max3,coef_last_outbound_dist_dest_gt1_max3,coef_last_outbound_dist_dest_gt1_max3,coef_last_outbound_dist_dest_gt1_max3 first inbound trip,"@(df.trip_num == 1) & (1 - df.outbound) & (_od_DIST > 1)",coef_first_inbound_dist_poe_gt_1mi,coef_first_inbound_dist_poe_gt_1mi,coef_first_inbound_dist_poe_gt_1mi,coef_first_inbound_dist_poe_gt_1mi,coef_first_inbound_dist_poe_gt_1mi last inbound intermediate stop,"@(df.trip_num == _stops_on_leg) & (1 - df.outbound) & (_dp_DIST > 1)",coef_last_inbound_dist_dest_gt_1mi,coef_last_inbound_dist_dest_gt_1mi,coef_last_inbound_dist_dest_gt_1mi,coef_last_inbound_dist_dest_gt_1mi,coef_last_inbound_dist_dest_gt_1mi -last inbound intermediate stop,"@(df.trip_num == _stops_on_leg) & (1 - df.outbound) * np.clip(_dp_DIST - 1, 0, 3)",coef_last_inbound_dist_dest_gt1_max3,coef_last_inbound_dist_dest_gt1_max3,coef_last_inbound_dist_dest_gt1_max3,coef_last_inbound_dist_dest_gt1_max3,coef_last_inbound_dist_dest_gt1_max3 \ No newline at end of file +last inbound intermediate stop,"@(df.trip_num == _stops_on_leg) & (1 - df.outbound) * np.clip(_dp_DIST - 1, 0, 3)",coef_last_inbound_dist_dest_gt1_max3,coef_last_inbound_dist_dest_gt1_max3,coef_last_inbound_dist_dest_gt1_max3,coef_last_inbound_dist_dest_gt1_max3,coef_last_inbound_dist_dest_gt1_max3 diff --git a/activitysim/examples/prototype_sandag_xborder/configs/trip_destination_annotate_trips_preprocessor.csv b/activitysim/examples/prototype_sandag_xborder/configs/trip_destination_annotate_trips_preprocessor.csv index d7563cc825..430f04cfe0 100644 --- a/activitysim/examples/prototype_sandag_xborder/configs/trip_destination_annotate_trips_preprocessor.csv +++ b/activitysim/examples/prototype_sandag_xborder/configs/trip_destination_annotate_trips_preprocessor.csv @@ -5,3 +5,4 @@ Description,Target,Expression ,tour_leg_dest,"np.where(df.outbound,reindex(tours.destination, df.tour_id), reindex(tours.origin, df.tour_id))" ,_tod,"np.where(df.outbound,reindex_i(tours.start, df.tour_id), reindex_i(tours.end, df.tour_id))" ,trip_period,network_los.skim_time_period_label(_tod) +,purpose_index_num,"size_terms.get_cols(df.purpose)" diff --git a/activitysim/examples/prototype_sandag_xborder/configs/trip_destination_sample.csv b/activitysim/examples/prototype_sandag_xborder/configs/trip_destination_sample.csv index 029c9f7ba4..7d047771b3 100644 --- a/activitysim/examples/prototype_sandag_xborder/configs/trip_destination_sample.csv +++ b/activitysim/examples/prototype_sandag_xborder/configs/trip_destination_sample.csv @@ -1,6 +1,6 @@ Description,Expression,work,school,visit,shop,other -Size variable,"@np.log1p(size_terms.get(df.trip_dest, df.purpose))",1,1,1,1,1 -No attractions,"@size_terms.get(df.trip_dest, df.purpose) == 0",-999,-999,-999,-999,-999 +Size variable,"@np.log1p(size_terms.get(df.trip_dest, df.purpose)) # sharrow: np.log1p(size_terms['arry'])",1,1,1,1,1 +No attractions,"@size_terms.get(df.trip_dest, df.purpose) == 0 # sharrow: size_terms['arry'] == 0",-999,-999,-999,-999,-999 ,"_od_DIST@odt_skims['SOV_NT_M_DIST']",1,1,1,1,1 ,"_dp_DIST@dpt_skims['SOV_NT_M_DIST']",1,1,1,1,1 intermediate stops on half-tour,_stops_on_leg@df.trip_count - 1,1,1,1,1,1 diff --git a/activitysim/examples/prototype_sandag_xborder/configs_sharrow/settings.yaml b/activitysim/examples/prototype_sandag_xborder/configs_sharrow/settings.yaml new file mode 100644 index 0000000000..7ddeaad9cf --- /dev/null +++ b/activitysim/examples/prototype_sandag_xborder/configs_sharrow/settings.yaml @@ -0,0 +1,3 @@ +inherit_settings: True +sharrow: require +recode_pipeline_columns: True diff --git a/activitysim/examples/prototype_sandag_xborder/data/taz_xborder.csv b/activitysim/examples/prototype_sandag_xborder/data/taz_xborder.csv new file mode 100644 index 0000000000..4813009ea6 --- /dev/null +++ b/activitysim/examples/prototype_sandag_xborder/data/taz_xborder.csv @@ -0,0 +1,95 @@ +TAZ +1 +2 +4 +3093 +3098 +3112 +3115 +3129 +3131 +3134 +3135 +3148 +3176 +3184 +3208 +3252 +3259 +3261 +3272 +3274 +3277 +3278 +3281 +3286 +3289 +3290 +3314 +3315 +3316 +3317 +3339 +3340 +3342 +3343 +3347 +3363 +3375 +3376 +3378 +3383 +3395 +3396 +3397 +3398 +3399 +3400 +3402 +3410 +3412 +3433 +3437 +3442 +3460 +3464 +3493 +3497 +3498 +3501 +3502 +3505 +3519 +3525 +3536 +3540 +3548 +3553 +3574 +3590 +3592 +3595 +3597 +3598 +3619 +3626 +3640 +3654 +3655 +3661 +3670 +3687 +3698 +3704 +3705 +3713 +3715 +3729 +3746 +3761 +3796 +3822 +3849 +4831 +4989 +4993 diff --git a/activitysim/examples/prototype_sandag_xborder/test/configs/settings.yaml b/activitysim/examples/prototype_sandag_xborder/test/configs/settings.yaml index 32e44ac90d..12039e3ab8 100644 --- a/activitysim/examples/prototype_sandag_xborder/test/configs/settings.yaml +++ b/activitysim/examples/prototype_sandag_xborder/test/configs/settings.yaml @@ -23,4 +23,9 @@ output_tables: prefix: final_ sort: True tables: - - trips + - tablename: trips + decode_columns: + origin: land_use.zone_id + destination: land_use.zone_id + +recode_pipeline_columns: False diff --git a/activitysim/examples/prototype_sandag_xborder/test/configs_sharrow/settings.yaml b/activitysim/examples/prototype_sandag_xborder/test/configs_sharrow/settings.yaml new file mode 100644 index 0000000000..7ddeaad9cf --- /dev/null +++ b/activitysim/examples/prototype_sandag_xborder/test/configs_sharrow/settings.yaml @@ -0,0 +1,3 @@ +inherit_settings: True +sharrow: require +recode_pipeline_columns: True diff --git a/activitysim/examples/prototype_sandag_xborder/test/configs_single_process/settings.yaml b/activitysim/examples/prototype_sandag_xborder/test/configs_single_process/settings.yaml new file mode 100644 index 0000000000..7e203b9e29 --- /dev/null +++ b/activitysim/examples/prototype_sandag_xborder/test/configs_single_process/settings.yaml @@ -0,0 +1,3 @@ +inherit_settings: True +num_processes: 1 +multiprocess: False diff --git a/activitysim/examples/prototype_sandag_xborder/test/regress/final_trips_1_process.csv b/activitysim/examples/prototype_sandag_xborder/test/regress/final_trips_1_process.csv new file mode 100644 index 0000000000..0c7e140aec --- /dev/null +++ b/activitysim/examples/prototype_sandag_xborder/test/regress/final_trips_1_process.csv @@ -0,0 +1,41 @@ +trip_id,person_id,household_id,primary_purpose,trip_num,outbound,trip_count,destination,origin,failed,tour_id,purpose,depart,trip_mode,trip_mode_choice_logsum,atap,btap,path_set +52689,29512,20525,work,1,True,1,100629,123004,False,6586,work,28,DRIVEALONE,-2.292706145734724,,, +52693,29512,20525,work,1,False,1,123004,100629,False,6586,home,43,SHARED3,-2.385303702957729,,, +88249,19755,85270,other,1,True,1,107090,123003,False,11031,other,24,WALK,0.6298754510725724,,, +88253,19755,85270,other,1,False,1,123003,107090,False,11031,home,32,SHARED2,0.5131544309500926,,, +95817,73905,28840,shop,1,True,4,107090,123003,False,11977,other,29,DRIVEALONE,1.0469859837689066,,, +95818,73905,28840,shop,2,True,4,107090,107090,False,11977,other,30,WALK,1.0920565735181922,,, +95819,73905,28840,shop,3,True,4,100770,107090,False,11977,shop,33,SHARED2,-0.7449355007220748,,, +95820,73905,28840,shop,4,True,4,100773,100770,False,11977,shop,36,SHARED2,1.295966425303992,,, +95821,73905,28840,shop,1,False,2,100628,100773,False,11977,shop,36,SHARED2,1.1468706683346748,,, +95822,73905,28840,shop,2,False,2,123003,100628,False,11977,home,36,SHARED2,-0.8665412248006725,,, +359641,53300,105709,shop,1,True,1,100949,123003,False,44955,shop,24,SHARED2,-1.5409694122199264,,, +359645,53300,105709,shop,1,False,4,107090,100949,False,44955,visit,36,DRIVEALONE,-1.5234074063798364,,, +359646,53300,105709,shop,2,False,4,100702,107090,False,44955,shop,37,WALK_TRANSIT,-0.7920988004454081,1748.0,2485.0,set2 +359647,53300,105709,shop,3,False,4,107090,100702,False,44955,other,37,DRIVEALONE,-0.8957792679038603,,, +359648,53300,105709,shop,4,False,4,123003,107090,False,44955,home,37,WALK,1.9934609859791363,,, +466729,101126,85142,shop,1,True,1,101060,123003,False,58341,shop,18,SHARED2,-0.8545911451016768,,, +466733,101126,85142,shop,1,False,4,107090,101060,False,58341,visit,24,WALK_TRANSIT,-0.8640844000020386,2485.0,1857.0,set3 +466734,101126,85142,shop,2,False,4,100681,107090,False,58341,shop,25,WALK_TRANSIT,-0.9762391774809218,1612.0,2485.0,set2 +466735,101126,85142,shop,3,False,4,100721,100681,False,58341,shop,25,WALK,2.1072924707467933,,, +466736,101126,85142,shop,4,False,4,123003,100721,False,58341,home,26,SHARED3,-1.0332751175513328,,, +524585,60566,5596,shop,1,True,2,107090,123003,False,65573,other,15,DRIVEALONE,-0.18358955282699793,,, +524586,60566,5596,shop,2,True,2,100672,107090,False,65573,shop,26,DRIVEALONE,-2.0960685672156,,, +524589,60566,5596,shop,1,False,4,100803,100672,False,65573,shop,32,SHARED2,-0.07701003498109199,,, +524590,60566,5596,shop,2,False,4,101070,100803,False,65573,shop,32,SHARED2,-0.7578421279004207,,, +524591,60566,5596,shop,3,False,4,107090,101070,False,65573,other,33,DRIVEALONE,-2.8049190561746626,,, +524592,60566,5596,shop,4,False,4,123003,107090,False,65573,home,33,DRIVEALONE,-0.31379684339847,,, +586217,35438,45199,shop,1,True,2,107090,123003,False,73277,other,20,SHARED3,1.4766852765241836,,, +586218,35438,45199,shop,2,True,2,100771,107090,False,73277,shop,21,SHARED3,-0.4943991865784387,,, +586221,35438,45199,shop,1,False,1,123003,100771,False,73277,home,32,SHARED3,-0.5803721903487284,,, +660081,95667,12980,shop,1,True,1,107090,123003,False,82510,shop,20,DRIVEALONE,1.556158267368568,,, +660085,95667,12980,shop,1,False,3,100853,107090,False,82510,shop,46,DRIVEALONE,-1.2731968603317538,,, +660086,95667,12980,shop,2,False,3,101016,100853,False,82510,shop,47,WALK_TRANSIT,1.4417223596004096,1666.0,1660.0,set1 +660087,95667,12980,shop,3,False,3,123003,101016,False,82510,home,48,WALK_TRANSIT,-1.6156535001905643,2485.0,1682.0,set3 +670945,15965,35585,work,1,True,2,107090,123003,False,83868,other,19,DRIVEALONE,0.5387885339726378,,, +670946,15965,35585,work,2,True,2,100511,107090,False,83868,work,20,SHARED2,-1.4113710136199296,,, +670949,15965,35585,work,1,False,1,123003,100511,False,83868,home,36,SHARED2,-1.462693708255357,,, +739817,12419,106831,shop,1,True,1,107090,123003,False,92477,shop,19,WALK,2.000393598935397,,, +739821,12419,106831,shop,1,False,3,107090,107090,False,92477,visit,21,WALK,2.0261526103543575,,, +739822,12419,106831,shop,2,False,3,101060,107090,False,92477,shop,22,SHARED3,-0.9555192489330274,,, +739823,12419,106831,shop,3,False,3,123003,101060,False,92477,home,24,WALK_TRANSIT,-1.0341377905668787,2485.0,1977.0,set2 diff --git a/activitysim/examples/prototype_sandag_xborder/test/test_sandag_xborder.py b/activitysim/examples/prototype_sandag_xborder/test/test_sandag_xborder.py index afa29c29d5..261190248e 100644 --- a/activitysim/examples/prototype_sandag_xborder/test/test_sandag_xborder.py +++ b/activitysim/examples/prototype_sandag_xborder/test/test_sandag_xborder.py @@ -2,6 +2,7 @@ # See full license in LICENSE.txt. import os import subprocess +import sys import pandas as pd import pandas.testing as pdt @@ -15,7 +16,7 @@ def teardown_function(func): inject.reinject_decorated_tables() -def test_sandag_xborder(): +def _test_sandag_xborder(sharrow=False, mp=True): def example_path(dirname): resource = os.path.join("examples", "prototype_sandag_xborder", dirname) return pkg_resources.resource_filename("activitysim", resource) @@ -24,33 +25,53 @@ def test_path(dirname): return os.path.join(os.path.dirname(__file__), dirname) def regress(): - regress_trips_df = pd.read_csv(test_path("regress/final_trips.csv")) + if mp: + regress_trips_df = pd.read_csv(test_path("regress/final_trips.csv")) + else: + regress_trips_df = pd.read_csv( + test_path("regress/final_trips_1_process.csv") + ) final_trips_df = pd.read_csv(test_path("output/final_trips.csv")) pdt.assert_frame_equal(final_trips_df, regress_trips_df) file_path = os.path.join(os.path.dirname(__file__), "../simulation.py") - subprocess.run( - [ - "coverage", - "run", - "-a", - file_path, - "-c", - test_path("configs"), - "-c", - example_path("configs"), - "-d", - example_path("data"), - "-o", - test_path("output"), - ], - check=True, - ) + run_args = [file_path] + if sharrow: + run_args += ["-c", test_path("configs_sharrow")] + if not mp: + run_args += ["-c", test_path("configs_single_process")] + run_args += [ + "-c", + test_path("configs"), + "-c", + example_path("configs"), + "-d", + example_path("data"), + "-o", + test_path("output"), + ] + + if os.environ.get("GITHUB_ACTIONS") == "true": + subprocess.run(["coverage", "run", "-a"] + run_args, check=True) + else: + subprocess.run([sys.executable] + run_args, check=True) regress() +def test_sandag_xborder(): + _test_sandag_xborder(mp=False) + + +def test_sandag_xborder_mp(): + _test_sandag_xborder(mp=True) + + +def test_sandag_xborder_sharrow(): + _test_sandag_xborder(sharrow=True, mp=False) + + if __name__ == "__main__": test_sandag_xborder() diff --git a/activitysim/examples/prototype_semcog/configs/logging.yaml b/activitysim/examples/prototype_semcog/configs/logging.yaml index 6dfca578a4..657507ee6e 100755 --- a/activitysim/examples/prototype_semcog/configs/logging.yaml +++ b/activitysim/examples/prototype_semcog/configs/logging.yaml @@ -1,53 +1,70 @@ -# Config for logging -# ------------------ -# See http://docs.python.org/2.7/library/logging.config.html#configuration-dictionary-schema - -logging: - version: 1 - disable_existing_loggers: true - - - # Configuring the default (root) logger is highly recommended - root: - level: NOTSET - handlers: [console] - - loggers: - - activitysim: - level: DEBUG - handlers: [console, logfile] - propagate: false - - orca: - level: WARN - handlers: [console, logfile] - propagate: false - - handlers: - - logfile: - class: logging.FileHandler - filename: !!python/object/apply:activitysim.core.config.log_file_path ['activitysim.log'] - mode: w - formatter: fileFormatter - level: NOTSET - - console: - class: logging.StreamHandler - stream: ext://sys.stdout - formatter: simpleFormatter - level: NOTSET - - formatters: - - simpleFormatter: - class: logging.Formatter - # format: '%(levelname)s - %(name)s - %(message)s' - format: '%(levelname)s - %(message)s' - datefmt: '%d/%m/%Y %H:%M:%S' - - fileFormatter: - class: logging.Formatter - format: '%(asctime)s - %(levelname)s - %(name)s - %(message)s' - datefmt: '%d/%m/%Y %H:%M:%S' +# Config for logging +# ------------------ +# See http://docs.python.org/2.7/library/logging.config.html#configuration-dictionary-schema + +logging: + version: 1 + disable_existing_loggers: true + + + # Configuring the default (root) logger is highly recommended + root: + level: NOTSET + handlers: [console] + + loggers: + + activitysim: + level: DEBUG + handlers: [console, logfile] + propagate: false + + orca: + level: WARN + handlers: [console, logfile] + propagate: false + + filelock: + level: WARN + + sharrow: + level: INFO + + blib2to3: + level: WARN + + black: + level: WARN + + handlers: + + logfile: + class: logging.FileHandler + filename: !!python/object/apply:activitysim.core.config.log_file_path ['activitysim.log'] + mode: w + formatter: fileFormatter + level: NOTSET + + console: + class: logging.StreamHandler + stream: ext://sys.stdout + formatter: elapsedFormatter + level: NOTSET + + formatters: + + simpleFormatter: + class: logging.Formatter + # format: '%(levelname)s - %(name)s - %(message)s' + format: '%(levelname)s - %(message)s' + datefmt: '%d/%m/%Y %H:%M:%S' + + fileFormatter: + class: logging.Formatter + format: '%(asctime)s - %(levelname)s - %(name)s - %(message)s' + datefmt: '%d/%m/%Y %H:%M:%S' + + elapsedFormatter: + (): activitysim.core.tracing.ElapsedTimeFormatter + format: '[{elapsedTime}] {levelname:s}: {message:s}' + style: '{' diff --git a/activitysim/examples/prototype_semcog/configs/mandatory_tour_frequency.csv b/activitysim/examples/prototype_semcog/configs/mandatory_tour_frequency.csv index 848bbf77aa..51094ee822 100755 --- a/activitysim/examples/prototype_semcog/configs/mandatory_tour_frequency.csv +++ b/activitysim/examples/prototype_semcog/configs/mandatory_tour_frequency.csv @@ -25,10 +25,10 @@ util_can_walk_to_work_retired,Can walk to work - Retired interaction,(ptype == 5 util_can_walk_to_school_univ,Can walk to school - University student interaction,(ptype == 3) & (distance_to_school < 3),,,,coef_can_walk_to_work_school2, util_can_walk_to_school_driving_age_child,Can walk to school - Driving-age child interaction,(ptype == 6) & (distance_to_school < 3),,,,coef_can_walk_to_work_school2, util_can_walk_to_school_pre_driving_age_child,Can walk to school - Pre-driving age child who is in school interaction,(ptype == 7) & (distance_to_school < 3),,,,coef_can_walk_to_work_school2, -util_can_walk_to_work_or_school_ft,Can walk to work or school - Full-time worker interaction,(ptype == 1) & (distance_to_work < 3 | distance_to_school < 3),,,,,coef_can_walk_to_work_and_school -util_can_walk_to_work_or_school_pt,Can walk to work or school - Part-time worker interaction,(ptype == 2) & (distance_to_work < 3 | distance_to_school < 3),,,,,coef_can_walk_to_work_and_school -util_can_walk_to_work_or_school_univ,Can walk to work or school - University student interaction,(ptype == 3) & (distance_to_work < 3 | distance_to_school < 3),,,,,coef_can_walk_to_work_and_school -util_can_walk_to_work_or_school_driving_age_child,Can walk to work or school - Driving-age child interaction,(ptype == 6) & (distance_to_work < 3 | distance_to_school < 3),,,,,coef_can_walk_to_work_and_school +util_can_walk_to_work_or_school_ft,Can walk to work or school - Full-time worker interaction,(ptype == 1) & ((distance_to_work < 3) | (distance_to_school < 3)),,,,,coef_can_walk_to_work_and_school +util_can_walk_to_work_or_school_pt,Can walk to work or school - Part-time worker interaction,(ptype == 2) & ((distance_to_work < 3) | (distance_to_school < 3)),,,,,coef_can_walk_to_work_and_school +util_can_walk_to_work_or_school_univ,Can walk to work or school - University student interaction,(ptype == 3) & ((distance_to_work < 3) | (distance_to_school < 3)),,,,,coef_can_walk_to_work_and_school +util_can_walk_to_work_or_school_driving_age_child,Can walk to work or school - Driving-age child interaction,(ptype == 6) & ((distance_to_work < 3) | (distance_to_school < 3)),,,,,coef_can_walk_to_work_and_school util_round_trip_auto_time_to_work_ft,Round trip auto time to work - Full-time worker interaction,(ptype == 1) * roundtrip_auto_time_to_work,,coef_round_trip_auto_time_to_work_work2,,,coef_round_trip_auto_time_to_work_school2 util_round_trip_auto_time_to_work_pt,Round trip auto time to work - Part-time worker interaction,(ptype == 2) * roundtrip_auto_time_to_work,,coef_round_trip_auto_time_to_work_work2,,,coef_round_trip_auto_time_to_work_school2 util_round_trip_auto_time_to_work_univ,Round trip auto time to work - University student interaction,(ptype == 3) * roundtrip_auto_time_to_work,,coef_round_trip_auto_time_to_work_work2,,,coef_round_trip_auto_time_to_work_school2 diff --git a/activitysim/examples/prototype_semcog/configs/settings.yaml b/activitysim/examples/prototype_semcog/configs/settings.yaml index 3d1bb870fc..2aeff430a2 100755 --- a/activitysim/examples/prototype_semcog/configs/settings.yaml +++ b/activitysim/examples/prototype_semcog/configs/settings.yaml @@ -1,156 +1,175 @@ -# input tables -# -# activitysim uses "well-known" index and foreign key names for imported tables (e.g. households, persons, land_use) -# as well as for created tables (tours, joint_tour_participants, trips) -# e.g. the households table must have an index column 'household_id' and the foreign key to households in the -# persons table is also household_id. This naming convention allows activitysim to intuit the relationship -# between tables - for instance, to ensure that multiprocess slicing includes all the persons, tours, and trips -# in the same subprocess pipeline. The same strategy is also when chunking choosers, and to support tracing by -# household_id. -# -# the input_table_list index_col directive instructs activitysim to set the imported table index to zone_id -# you cannot change the well-known name of the index by modifying this directive. However, if your input file -# has a different id column name, you can rename it to the required index name with the rename_columns directive. -# In the settings below, the 'TAZ' column in the imported table is renamed 'zone_id' in the rename_columns settings. -# -# input tables -input_table_list: - # - # households (table index 'household_id') - # - - tablename: households - filename: households.csv - index_col: household_id - rename_columns: - persons: hhsize - cars: auto_ownership - zone_id: home_zone_id - # - # persons (table index 'person_id') - # - - tablename: persons - filename: persons.csv - index_col: person_id - # - # land_use (table index 'zone_id') - # - - tablename: land_use - filename: land_use.csv - index_col: zone_id - rename_columns: - ZONE: zone_id - -# convert input CSVs to HDF5 format and save to outputs directory -# create_input_store: True - -# number of households to simulate -households_sample_size: 100 -# simulate all households -#households_sample_size: 0 - -#hh_ids: override_hh_ids.csv - -chunk_size: 0 - -# assume enough RAM to not chunk -chunk_training_mode: disabled - -# set false to disable variability check in simple_simulate and interaction_simulate -check_for_variability: False - -# - shadow pricing global switches - -# turn shadow_pricing on and off for all models (e.g. school and work) -# shadow pricing is deprecated for less than full samples -# see shadow_pricing.yaml for additional settings -use_shadow_pricing: False - - -# - tracing - -# trace household id; comment out or leave empty for no trace -# households with all tour types -# [ 728370 1234067 1402924 1594625 1595333 1747572 1896849 1931818 2222690 2344951 2677154] -trace_hh_id: 1311364 - -# trace origin, destination in accessibility calculation; comment out or leave empty for no trace -# trace_od: [5, 11] -trace_od: - - -models: - - initialize_landuse - - initialize_households - - compute_accessibility - - work_from_home - - add_size_tables - - school_location - - workplace_location - - transit_pass_subsidy - - transit_pass_ownership - - auto_ownership_simulate - - free_parking - - telecommute_frequency - - cdap_simulate - - mandatory_tour_frequency - - mandatory_tour_scheduling - - joint_tour_frequency - - joint_tour_composition - - joint_tour_participation - - joint_tour_destination - - joint_tour_scheduling - - non_mandatory_tour_frequency - - non_mandatory_tour_destination - - non_mandatory_tour_scheduling - - tour_mode_choice_simulate - - atwork_subtour_frequency - - atwork_subtour_destination - - atwork_subtour_scheduling - - atwork_subtour_mode_choice - - stop_frequency - - trip_purpose - - trip_destination - - trip_purpose_and_destination - - trip_scheduling - - trip_mode_choice - - write_data_dictionary - - track_skim_usage - - write_tables - - write_trip_matrices - -# to resume after last successful checkpoint, specify resume_after: _ -resume_after: - -output_tables: - h5_store: False - action: include - prefix: final_ - tables: - - checkpoints - - accessibility - - land_use - - households - - persons - - tours - - trips - - joint_tour_participants - -# area_types less than this are considered urban -urban_threshold: 4 -cbd_threshold: 2 -rural_threshold: 6 - -# - value of time - -# value_of_time = lognormal(np.log(median_value_of_time * mu), sigma).clip(min_vot, max_vot) - -min_value_of_time: 1 -max_value_of_time: 50 -distributed_vot_mu: 0.684 -distributed_vot_sigma: 0.85 - -household_median_value_of_time: - 1: 6.01 - 2: 8.81 - 3: 10.44 - 4: 12.86 +# input tables +# +# activitysim uses "well-known" index and foreign key names for imported tables (e.g. households, persons, land_use) +# as well as for created tables (tours, joint_tour_participants, trips) +# e.g. the households table must have an index column 'household_id' and the foreign key to households in the +# persons table is also household_id. This naming convention allows activitysim to intuit the relationship +# between tables - for instance, to ensure that multiprocess slicing includes all the persons, tours, and trips +# in the same subprocess pipeline. The same strategy is also when chunking choosers, and to support tracing by +# household_id. +# +# the input_table_list index_col directive instructs activitysim to set the imported table index to zone_id +# you cannot change the well-known name of the index by modifying this directive. However, if your input file +# has a different id column name, you can rename it to the required index name with the rename_columns directive. +# In the settings below, the 'TAZ' column in the imported table is renamed 'zone_id' in the rename_columns settings. +# +# input tables +input_table_list: + # + # households (table index 'household_id') + # + - tablename: households + filename: households.csv + index_col: household_id + rename_columns: + persons: hhsize + cars: auto_ownership + zone_id: home_zone_id + recode_columns: + home_zone_id: land_use.zone_id + # + # persons (table index 'person_id') + # + - tablename: persons + filename: persons.csv + index_col: person_id + # + # land_use (table index 'zone_id') + # + - tablename: land_use + filename: land_use.csv + index_col: zone_id + rename_columns: + ZONE: zone_id + recode_columns: + zone_id: zero-based + +# convert input CSVs to HDF5 format and save to outputs directory +# create_input_store: True + +# number of households to simulate +households_sample_size: 100 +# simulate all households +#households_sample_size: 0 + +#hh_ids: override_hh_ids.csv + +chunk_size: 0 + +# assume enough RAM to not chunk +chunk_training_mode: disabled + +# set false to disable variability check in simple_simulate and interaction_simulate +check_for_variability: False + +# - shadow pricing global switches + +# turn shadow_pricing on and off for all models (e.g. school and work) +# shadow pricing is deprecated for less than full samples +# see shadow_pricing.yaml for additional settings +use_shadow_pricing: False + + +# - tracing + +# trace household id; comment out or leave empty for no trace +# households with all tour types +# [ 728370 1234067 1402924 1594625 1595333 1747572 1896849 1931818 2222690 2344951 2677154] +trace_hh_id: 1311364 + +# trace origin, destination in accessibility calculation; comment out or leave empty for no trace +# trace_od: [5, 11] +trace_od: + + +models: + - initialize_landuse + - initialize_households + - compute_accessibility + - work_from_home + - add_size_tables + - school_location + - workplace_location + - transit_pass_subsidy + - transit_pass_ownership + - auto_ownership_simulate + - free_parking + - telecommute_frequency + - cdap_simulate + - mandatory_tour_frequency + - mandatory_tour_scheduling + - joint_tour_frequency + - joint_tour_composition + - joint_tour_participation + - joint_tour_destination + - joint_tour_scheduling + - non_mandatory_tour_frequency + - non_mandatory_tour_destination + - non_mandatory_tour_scheduling + - tour_mode_choice_simulate + - atwork_subtour_frequency + - atwork_subtour_destination + - atwork_subtour_scheduling + - atwork_subtour_mode_choice + - stop_frequency + - trip_purpose + - trip_destination + - trip_purpose_and_destination + - trip_scheduling + - trip_mode_choice + - write_data_dictionary + - track_skim_usage + - write_tables + - write_trip_matrices + +# to resume after last successful checkpoint, specify resume_after: _ +resume_after: + +output_tables: + h5_store: False + action: include + prefix: final_ + tables: + - checkpoints + - accessibility + - tablename: land_use + decode_columns: + zone_id: land_use.zone_id + - tablename: households + decode_columns: + home_zone_id: land_use.zone_id + - tablename: persons + decode_columns: + home_zone_id: land_use.zone_id + school_zone_id: nonnegative | land_use.zone_id + workplace_zone_id: nonnegative | land_use.zone_id + - tablename: tours + decode_columns: + origin: land_use.zone_id + destination: land_use.zone_id + - tablename: trips + decode_columns: + origin: land_use.zone_id + destination: land_use.zone_id + - joint_tour_participants + +# area_types less than this are considered urban +urban_threshold: 4 +cbd_threshold: 2 +rural_threshold: 6 + +# - value of time + +# value_of_time = lognormal(np.log(median_value_of_time * mu), sigma).clip(min_vot, max_vot) + +min_value_of_time: 1 +max_value_of_time: 50 +distributed_vot_mu: 0.684 +distributed_vot_sigma: 0.85 + +household_median_value_of_time: + 1: 6.01 + 2: 8.81 + 3: 10.44 + 4: 12.86 + diff --git a/activitysim/examples/prototype_semcog/configs/tour_mode_choice.csv b/activitysim/examples/prototype_semcog/configs/tour_mode_choice.csv index 14e13a802f..dfeb0a526a 100755 --- a/activitysim/examples/prototype_semcog/configs/tour_mode_choice.csv +++ b/activitysim/examples/prototype_semcog/configs/tour_mode_choice.csv @@ -8,7 +8,7 @@ util_DRIVEALONE_In_vehicle_time,DRIVEALONE - In-vehicle time,@odt_skims['SOV_TIM util_DRIVEALONE_Terminal_time,DRIVEALONE - Terminal time,@2 * walktimeshort_multiplier * df.terminal_time,0,,,,,,,,,,,,,,,,, util_DRIVEALONE_Operating_cost_,DRIVEALONE - Operating cost ,@ivt_cost_multiplier * df.ivot * costPerMile * (odt_skims['SOV_DIST'] + dot_skims['SOV_DIST']),coef_ivt,,,,,,,,,,,,,,,,, util_DRIVEALONE_Parking_cost_,DRIVEALONE - Parking cost ,@ivt_cost_multiplier * df.ivot * df.daily_parking_cost,coef_ivt,,,,,,,,,,,,,,,,, -util_DRIVEALONE_Bridge_toll_,DRIVEALONE - Bridge toll ,@ivt_cost_multiplier * df.ivot * (odt_skims['SOV_BTOLL'] + dot_skims['SOV_BTOLL']),0,,,,,,,,,,,,,,,,, +#util_DRIVEALONE_Bridge_toll_,DRIVEALONE - Bridge toll ,@ivt_cost_multiplier * df.ivot * (odt_skims['SOV_BTOLL'] + dot_skims['SOV_BTOLL']),0,,,,,,,,,,,,,,,,, util_DRIVEALONE_Person_is_between_16_and_19_years_old,DRIVEALONE - Person is between 16 and 19 years old,@(df.age >= 16) & (df.age <= 19),coef_age1619_da_multiplier,,,,,,,,,,,,,,,,, #Shared_ride_2,#Shared ride 2,,,,,,,,,,,,,,,,,,, util_SHARED2_Unavailable,SHARED2 - Unavailable,hov2_available == False,,-999,,,,,,,,,,,,,,,, @@ -17,7 +17,7 @@ util_SHARED2_In_vehicle_time,SHARED2 - In-vehicle time,@(odt_skims['HOV2_TIME'] util_SHARED2_Terminal_time,SHARED2 - Terminal time,@2 * walktimeshort_multiplier * df.terminal_time,,0,,,,,,,,,,,,,,,, util_SHARED2_Operating_cost,SHARED2 - Operating cost,@ivt_cost_multiplier * df.ivot * costPerMile * (odt_skims['HOV2_DIST'] + dot_skims['HOV2_DIST']),,coef_ivt,,,,,,,,,,,,,,,, util_SHARED2_Parking_cost,SHARED2 - Parking cost,@ivt_cost_multiplier * df.ivot * df.daily_parking_cost / costShareSr2,,coef_ivt,,,,,,,,,,,,,,,, -util_SHARED2_Bridge_toll,SHARED2 - Bridge toll,@ivt_cost_multiplier * df.ivot * (odt_skims['HOV2_BTOLL'] + dot_skims['HOV2_BTOLL']) / costShareSr2,,0,,,,,,,,,,,,,,,, +#util_SHARED2_Bridge_toll,SHARED2 - Bridge toll,@ivt_cost_multiplier * df.ivot * (odt_skims['HOV2_BTOLL'] + dot_skims['HOV2_BTOLL']) / costShareSr2,,0,,,,,,,,,,,,,,,, util_SHARED2_One_person_household,SHARED2 - One person household,@(df.hhsize == 1),,coef_hhsize1_sr_multiplier,,,,,,,,,,,,,,,, util_SHARED2_Two_person_household,SHARED2 - Two person household,@(df.hhsize == 2),,coef_hhsize2_sr_multiplier,,,,,,,,,,,,,,,, util_SHARED2_Person_is_16_years_old_or_older,SHARED2 - Person is 16 years old or older,@(df.age >= 16),,coef_age16p_sr_multiplier,,,,,,,,,,,,,,,, @@ -27,7 +27,7 @@ util_SHARED3_In_vehicle_time,SHARED3 - In-vehicle time,@(odt_skims['HOV3_TIME'] util_SHARED3_Terminal_time,SHARED3 - Terminal time,@2 * walktimeshort_multiplier * df.terminal_time,,,0,,,,,,,,,,,,,,, util_SHARED3_Operating_cost,SHARED3 - Operating cost,@ivt_cost_multiplier * df.ivot * costPerMile * (odt_skims['HOV3_DIST'] + dot_skims['HOV3_DIST']),,,coef_ivt,,,,,,,,,,,,,,, util_SHARED3_Parking_cost,SHARED3 - Parking cost,@ivt_cost_multiplier * df.ivot * df.daily_parking_cost / costShareSr3,,,coef_ivt,,,,,,,,,,,,,,, -util_SHARED3_Bridge_toll,SHARED3 - Bridge toll,@ivt_cost_multiplier * df.ivot * (odt_skims['HOV3_BTOLL'] + dot_skims['HOV3_BTOLL']) / costShareSr3,,,0,,,,,,,,,,,,,,, +#util_SHARED3_Bridge_toll,SHARED3 - Bridge toll,@ivt_cost_multiplier * df.ivot * (odt_skims['HOV3_BTOLL'] + dot_skims['HOV3_BTOLL']) / costShareSr3,,,0,,,,,,,,,,,,,,, util_SHARED3_One_person_household,SHARED3 - One person household,@(df.hhsize == 1),,,coef_hhsize1_sr_multiplier,,,,,,,,,,,,,,, util_SHARED3_Two_person_household,SHARED3 - Two person household,@(df.hhsize == 2),,,coef_hhsize2_sr_multiplier,,,,,,,,,,,,,,, util_SHARED3_Person_is_16_years_old_or_older,SHARED3 - Person is 16 years old or older,@(df.age >= 16),,,coef_age16p_sr_multiplier,,,,,,,,,,,,,,, diff --git a/activitysim/examples/prototype_semcog/configs/trip_destination.csv b/activitysim/examples/prototype_semcog/configs/trip_destination.csv index 2d9307ab58..0e975f3ade 100755 --- a/activitysim/examples/prototype_semcog/configs/trip_destination.csv +++ b/activitysim/examples/prototype_semcog/configs/trip_destination.csv @@ -1,6 +1,6 @@ Description,Expression,work,univ,school,escort,shopping,eatout,othmaint,social,othdiscr,atwork -size term,"@np.log1p(size_terms.get(df.dest_taz, df.purpose))",1,1,1,1,1,1,1,1,1,1 -no attractions,"@size_terms.get(df.dest_taz, df.purpose) == 0",-999,-999,-999,-999,-999,-999,-999,-999,-999,-999 +size term,"@np.log1p(size_terms.get(df.dest_taz, df.purpose)) # sharrow: np.log1p(size_terms['arry'])",1,1,1,1,1,1,1,1,1,1 +no attractions,"@size_terms.get(df.dest_taz, df.purpose) == 0 # sharrow: size_terms['arry'] == 0",-999,-999,-999,-999,-999,-999,-999,-999,-999,-999 #stop zone CBD area type,"@reindex(land_use.AreaType, df.dest_taz) < setting('cbd_threshold')",,,,,,,,,, distance (calibration adjustment individual - inbound),@(~df.is_joint & ~df.outbound) * (od_skims['DIST'] + dp_skims['DIST']),-0.049725916,-0.0613,-0.1056,-0.1491,-0.1192,-0.1029,-0.0962,-0.1329,-0.126172224,-0.122334597 distance (calibration adjustment individual - outbound),@(~df.is_joint & df.outbound) * (od_skims['DIST'] + dp_skims['DIST']),0.147813279,-0.0613,-0.1056,-0.1491,-0.1192,-0.1029,-0.0962,-0.1329,-0.126172224,-0.122334597 diff --git a/activitysim/examples/prototype_semcog/configs/trip_destination_annotate_trips_preprocessor.csv b/activitysim/examples/prototype_semcog/configs/trip_destination_annotate_trips_preprocessor.csv index 1a1afb0748..9f2d502d3a 100755 --- a/activitysim/examples/prototype_semcog/configs/trip_destination_annotate_trips_preprocessor.csv +++ b/activitysim/examples/prototype_semcog/configs/trip_destination_annotate_trips_preprocessor.csv @@ -7,4 +7,7 @@ Description,Target,Expression #,,not needed as school is not chosen as an intermediate trip destination #,_grade_school,"(df.primary_purpose == 'school') & reindex(persons.is_gradeschool, df.person_id)" #,size_segment,"df.primary_purpose.where(df.primary_purpose != 'school', np.where(_grade_school,'gradeschool', 'highschool'))" -,tour_leg_dest,"np.where(df.outbound,reindex(tours.destination, df.tour_id), reindex(tours.origin, df.tour_id))" \ No newline at end of file +,purpose_index_num,"size_terms.get_cols(df.purpose)" +,tour_mode_is_walk,"reindex(tours.tour_mode, df.tour_id)=='WALK'" +,tour_mode_is_bike,"reindex(tours.tour_mode, df.tour_id)=='BIKE'" +,tour_leg_dest,"np.where(df.outbound,reindex(tours.destination, df.tour_id), reindex(tours.origin, df.tour_id)).astype(int)" diff --git a/activitysim/examples/prototype_semcog/configs/trip_destination_sample.csv b/activitysim/examples/prototype_semcog/configs/trip_destination_sample.csv index 6a0a53480e..692f2bd6eb 100755 --- a/activitysim/examples/prototype_semcog/configs/trip_destination_sample.csv +++ b/activitysim/examples/prototype_semcog/configs/trip_destination_sample.csv @@ -1,13 +1,13 @@ Description,Expression,work,univ,school,escort,shopping,eatout,othmaint,social,othdiscr,atwork ,_od_DIST@od_skims['DIST'],1,1,1,1,1,1,1,1,1,1 ,_dp_DIST@dp_skims['DIST'],1,1,1,1,1,1,1,1,1,1 -Not available if walk tour not within walking distance,@(df.tour_mode=='WALK') & (od_skims['DISTWALK'] > max_walk_distance),-999,-999,-999,-999,-999,-999,-999,-999,-999,-999 -Not available if walk tour not within walking distance,@(df.tour_mode=='WALK') & (dp_skims['DISTWALK'] > max_walk_distance),-999,-999,-999,-999,-999,-999,-999,-999,-999,-999 -Not available if bike tour not within biking distance,@(df.tour_mode=='BIKE') & (od_skims['DISTBIKE'] > max_bike_distance),-999,-999,-999,-999,-999,-999,-999,-999,-999,-999 -Not available if bike tour not within biking distance,@(df.tour_mode=='BIKE') & (dp_skims['DISTBIKE'] > max_bike_distance),-999,-999,-999,-999,-999,-999,-999,-999,-999,-999 +Not available if walk tour not within walking distance,@(df.tour_mode_is_walk) & (od_skims['DISTWALK'] > max_walk_distance),-999,-999,-999,-999,-999,-999,-999,-999,-999,-999 +Not available if walk tour not within walking distance,@(df.tour_mode_is_walk) & (dp_skims['DISTWALK'] > max_walk_distance),-999,-999,-999,-999,-999,-999,-999,-999,-999,-999 +Not available if bike tour not within biking distance,@(df.tour_mode_is_bike) & (od_skims['DISTBIKE'] > max_bike_distance),-999,-999,-999,-999,-999,-999,-999,-999,-999,-999 +Not available if bike tour not within biking distance,@(df.tour_mode_is_bike) & (dp_skims['DISTBIKE'] > max_bike_distance),-999,-999,-999,-999,-999,-999,-999,-999,-999,-999 #If transit tour is not in walk sub-zone it must be walkable,,,,,,,,,,, -size term,"@np.log1p(size_terms.get(df.dest_taz, df.purpose))",1,1,1,1,1,1,1,1,1,1 -no attractions,"@size_terms.get(df.dest_taz, df.purpose) == 0",-999,-999,-999,-999,-999,-999,-999,-999,-999,-999 +size term,"@np.log1p(size_terms.get(df.dest_taz, df.purpose)) # sharrow: np.log1p(size_terms['arry'])",1,1,1,1,1,1,1,1,1,1 +no attractions,"@size_terms.get(df.dest_taz, df.purpose) == 0 # sharrow: size_terms['arry'] == 0",-999,-999,-999,-999,-999,-999,-999,-999,-999,-999 #stop zone CBD area type,"@reindex(land_use.AreaType, df.dest_taz) < setting('cbd_threshold')",,,,,,,,,, distance (calibration adjustment individual - inbound),@(~df.is_joint & ~df.outbound) * (_od_DIST + _dp_DIST),-0.049725916,-0.0613,-0.1056,-0.1491,-0.1192,-0.1029,-0.0962,-0.1329,-0.126172224,-0.122334597 distance (calibration adjustment individual - outbound),@(~df.is_joint & df.outbound) * (_od_DIST + _dp_DIST),0.147813279,-0.0613,-0.1056,-0.1491,-0.1192,-0.1029,-0.0962,-0.1329,-0.126172224,-0.122334597 @@ -15,4 +15,4 @@ distance (calibration adjustment joint),@df.is_joint * (_od_DIST + _dp_DIST),0,0 stop proximity to home (outbound),@df.outbound * _od_DIST,-0.38,0,0,0,0,0,0,0,0,0 stop proximity to home (inbound),@~df.outbound * _od_DIST,-0.15,0,0,0,0,0,0,0,0,0 stop proximity to main destination (outbound),@df.outbound * _dp_DIST,-0.26,,,,,,,,, -stop proximity to main destination (inbound),@~df.outbound * _od_DIST,0,,,,,,,,, +stop proximity to main destination (inbound),@~df.outbound * _od_DIST,0,,,,,,,,, \ No newline at end of file diff --git a/activitysim/examples/prototype_semcog/configs/work_from_home.csv b/activitysim/examples/prototype_semcog/configs/work_from_home.csv index 11cfe76b4c..b5afec73fc 100755 --- a/activitysim/examples/prototype_semcog/configs/work_from_home.csv +++ b/activitysim/examples/prototype_semcog/configs/work_from_home.csv @@ -2,9 +2,9 @@ Label,Description,Expression,work_at_home,work_away_from_home util_work_from_home_constant,Constant for Working from home,1,coef_work_from_home_constant,0 util_full_time_worker,Full time worker (1 if true),@df.pemploy==PEMPLOY_FULL,coef_full_time_worker,0 util_female_worker,Female Worker,@df.sex==2,coef_female_worker,0 -util_female_worker_preschool_child,Female worker with a Preschool Child in Household,"@df.sex==2 & other_than(df.household_id, df.ptype == PTYPE_SCHOOL)",coef_female_worker_preschool_child,0 +util_female_worker_preschool_child,Female worker with a Preschool Child in Household,female_worker_with_preschool_child,coef_female_worker_preschool_child,0 util_access_to_workplaces,Accessibility to workplaces of the home mgra,@df.auPkTotal,coef_access_to_workplaces,0 -util_non_working_adult_in_hh,Presence of Non Working Adult in the Household,"@other_than(df.household_id, df.ptype == PTYPE_NONWORK)",coef_non_working_adult_in_hh,0 +util_non_working_adult_in_hh,Presence of Non Working Adult in the Household,nonworking_adult_in_hh,coef_non_working_adult_in_hh,0 util_education_ba_plus,Education Level Bachelors or higher degree,0,coef_education_ba_plus,0 util_low_income,Household income Less than 30K,@df.income < 30000,coef_low_income,0 util_age_lt_35,Age Group - Less than 35 years,@df.age < 35,coef_age_lt_35,0 diff --git a/activitysim/examples/prototype_semcog/configs/work_from_home.yaml b/activitysim/examples/prototype_semcog/configs/work_from_home.yaml index b94c138836..623c57b355 100755 --- a/activitysim/examples/prototype_semcog/configs/work_from_home.yaml +++ b/activitysim/examples/prototype_semcog/configs/work_from_home.yaml @@ -1,19 +1,23 @@ - -SPEC: work_from_home.csv -COEFFICIENTS: work_from_home_coeffs.csv - -#LOGIT_TYPE: NL -LOGIT_TYPE: MNL - -WORK_FROM_HOME_ALT: 0 - -# boolean column to filter choosers (True means keep) -CHOOSER_FILTER_COLUMN_NAME: is_worker - -# iterative what-if analysis example -# omit these settings to not iterate -# WORK_FROM_HOME_ITERATIONS: 3 -# WORK_FROM_HOME_CHOOSER_FILTER: is_worker -# WORK_FROM_HOME_TARGET_PERCENT: 0.1 -# WORK_FROM_HOME_TARGET_PERCENT_TOLERANCE: 0.01 -# WORK_FROM_HOME_COEFFICIENT_CONSTANT: coef_work_from_home_constant \ No newline at end of file + +SPEC: work_from_home.csv +COEFFICIENTS: work_from_home_coeffs.csv + +#LOGIT_TYPE: NL +LOGIT_TYPE: MNL + +WORK_FROM_HOME_ALT: 0 + +# boolean column to filter choosers (True means keep) +CHOOSER_FILTER_COLUMN_NAME: is_worker + +# iterative what-if analysis example +# omit these settings to not iterate +# WORK_FROM_HOME_ITERATIONS: 3 +# WORK_FROM_HOME_CHOOSER_FILTER: is_worker +# WORK_FROM_HOME_TARGET_PERCENT: 0.1 +# WORK_FROM_HOME_TARGET_PERCENT_TOLERANCE: 0.01 +# WORK_FROM_HOME_COEFFICIENT_CONSTANT: coef_work_from_home_constant + +preprocessor: + SPEC: work_from_home_annotate_persons_preprocessor + DF: persons diff --git a/activitysim/examples/prototype_semcog/configs/work_from_home_annotate_persons_preprocessor.csv b/activitysim/examples/prototype_semcog/configs/work_from_home_annotate_persons_preprocessor.csv new file mode 100644 index 0000000000..1eb7f1b2e0 --- /dev/null +++ b/activitysim/examples/prototype_semcog/configs/work_from_home_annotate_persons_preprocessor.csv @@ -0,0 +1,3 @@ +Description,Target,Expression +,female_worker_with_preschool_child,"df.sex==2 & other_than(df.household_id, df.ptype == PTYPE_SCHOOL)" +,nonworking_adult_in_hh,"other_than(df.household_id, df.ptype == PTYPE_NONWORK)" diff --git a/activitysim/examples/prototype_semcog/extensions/__init__.py b/activitysim/examples/prototype_semcog/extensions/__init__.py deleted file mode 100644 index 0b9ada0334..0000000000 --- a/activitysim/examples/prototype_semcog/extensions/__init__.py +++ /dev/null @@ -1,6 +0,0 @@ -from . import ( - telecommute_frequency, - transit_pass_ownership, - transit_pass_subsidy, - work_from_home, -) diff --git a/activitysim/examples/prototype_semcog/simulation.py b/activitysim/examples/prototype_semcog/simulation.py index d53f5d25c7..8313dd45e7 100755 --- a/activitysim/examples/prototype_semcog/simulation.py +++ b/activitysim/examples/prototype_semcog/simulation.py @@ -4,8 +4,6 @@ import argparse import sys -import extensions - from activitysim.cli.run import add_run_args, run if __name__ == "__main__": diff --git a/activitysim/preconfigure.py b/activitysim/preconfigure.py new file mode 100644 index 0000000000..73eb6902bd --- /dev/null +++ b/activitysim/preconfigure.py @@ -0,0 +1,14 @@ +import os + + +def main(): + from .cli.main import main + + os.environ["MKL_NUM_THREADS"] = "1" + os.environ["OMP_NUM_THREADS"] = "1" + os.environ["OPENBLAS_NUM_THREADS"] = "1" + os.environ["NUMBA_NUM_THREADS"] = "1" + os.environ["VECLIB_MAXIMUM_THREADS"] = "1" + os.environ["NUMEXPR_NUM_THREADS"] = "1" + + return main() diff --git a/activitysim/standalone/__init__.py b/activitysim/standalone/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/activitysim/standalone/compare.py b/activitysim/standalone/compare.py new file mode 100644 index 0000000000..024c720a3f --- /dev/null +++ b/activitysim/standalone/compare.py @@ -0,0 +1,319 @@ +import os +import warnings + +import altair as alt +import pandas as pd + +from .data_dictionary import check_data_dictionary +from .pipeline import load_checkpointed_tables + + +def load_pipelines(pipelines, tables=None, checkpoint_name=None): + """ + Parameters + ---------- + pipelines : Dict[Str, Path-like] + Mapping run name to path of pipeline file. + checkpoint : str + Name of checkpoint to load for all pipelines + """ + return { + key: load_checkpointed_tables( + pth, + tables=tables, + checkpoint_name=checkpoint_name, + )[1] + for key, pth in pipelines.items() + } + + +def load_final_tables(output_dirs, tables=None, index_cols=None): + result = {} + for key, pth in output_dirs.items(): + if not os.path.exists(pth): + warnings.warn(f"{key} directory does not exist: {pth}") + continue + result[key] = {} + for tname, tfile in tables.items(): + tpath = os.path.join(pth, tfile) + kwargs = {} + if index_cols is not None and tname in index_cols: + kwargs["index_col"] = index_cols[tname] + if os.path.exists(tpath): + result[key][tname] = pd.read_csv(tpath, **kwargs) + if len(result[key]) == 0: + # no tables were loaded, delete the entire group + del result[key] + return result + + +def compare_trip_mode_choice( + tablesets, title="Trip Mode Choice", grouping="primary_purpose" +): + + d = {} + groupings = [ + grouping, + ] + + for key, tableset in tablesets.items(): + df = ( + tableset["trips"] + .groupby(groupings + ["trip_mode"]) + .size() + .rename("n_trips") + .reset_index() + ) + df["share_trips"] = df["n_trips"] / df.groupby(groupings)["n_trips"].transform( + "sum" + ) + d[key] = df + + all_d = pd.concat(d, names=["source"]).reset_index() + + selection = alt.selection_multi( + fields=["trip_mode"], + bind="legend", + ) + + fig = ( + alt.Chart(all_d) + .mark_bar() + .encode( + color="trip_mode", + y=alt.Y("source", axis=alt.Axis(grid=False, title=""), sort=None), + x=alt.X( + "share_trips", + axis=alt.Axis(grid=False, labels=False, title="Mode Share"), + ), + row="primary_purpose", + opacity=alt.condition(selection, alt.value(1), alt.value(0.2)), + tooltip=[ + "trip_mode", + "source", + "n_trips", + alt.Tooltip("share_trips:Q", format=".2%"), + ], + ) + .add_selection( + selection, + ) + ) + + if title: + fig = fig.properties(title=title).configure_title( + fontSize=20, + anchor="start", + color="black", + ) + + return fig + + +def compare_trip_distance( + tablesets, + skims, + dist_skim_name, + otaz_col="origin", + dtaz_col="destination", + time_col="depart", + dist_bins=20, + grouping="primary_purpose", + title="Trip Length Distribution", + max_dist=None, +): + groupings = [grouping] + if not isinstance(skims, dict): + skims = {i: skims for i in tablesets.keys()} + + distances = {} + for key, tableset in tablesets.items(): + skim_dist = skims[key][[dist_skim_name]] + + zone_ids = tableset["land_use"].index + if ( + zone_ids.is_monotonic_increasing + and zone_ids[-1] == len(zone_ids) + zone_ids[0] - 1 + ): + offset = zone_ids[0] + looks = [ + tableset["trips"][otaz_col].rename("otaz") - offset, + tableset["trips"][dtaz_col].rename("dtaz") - offset, + ] + else: + remapper = dict(zip(zone_ids, pd.RangeIndex(len(zone_ids)))) + looks = [ + tableset["trips"][otaz_col].rename("otaz").apply(remapper.get), + tableset["trips"][dtaz_col].rename("dtaz").apply(remapper.get), + ] + if "time_period" in skim_dist.dims: + looks.append( + tableset["trips"][time_col] + .apply(skims[key].attrs["time_period_imap"].get) + .rename("time_period"), + ) + look = pd.concat(looks, axis=1) + distances[key] = skims[key][[dist_skim_name]].iat.df(look) + + if dist_bins is not None: + result = pd.concat(distances, names=["source"]) + if max_dist is not None: + result = result[result <= max_dist] + result = pd.cut(result.iloc[:, 0], dist_bins).to_frame() + distances = {k: result.loc[k] for k in tablesets.keys()} + + data = {} + for key, tableset in tablesets.items(): + data[key] = tableset["trips"].assign(**{"distance": distances[key]}) + + d = {} + for key, dat in data.items(): + df = ( + dat.groupby(groupings + ["distance"]) + .size() + .rename("n_trips") + .unstack("distance") + .fillna(0) + .stack() + .rename("n_trips") + .reset_index() + ) + df["share_trips"] = df["n_trips"] / df.groupby(groupings)["n_trips"].transform( + "sum" + ) + d[key] = df + + all_d = pd.concat(d, names=["source"]).reset_index() + all_d["distance"] = all_d["distance"].apply(lambda x: x.mid) + + fig = ( + alt.Chart(all_d) + .mark_line( + interpolate="monotone", + ) + .encode( + color="source", + y=alt.Y("share_trips", axis=alt.Axis(grid=False, title="")), + x=alt.X("distance", axis=alt.Axis(grid=False, title="Distance")), + # opacity=alt.condition(selection, alt.value(1), alt.value(0.2)), + # tooltip = ['trip_mode', 'source', 'n_trips', alt.Tooltip('share_trips:Q', format='.2%')], + facet=alt.Facet(grouping, columns=3), + strokeWidth="source", + ) + .properties( + width=200, + height=120, + ) + ) + + if title: + fig = fig.properties(title=title).configure_title( + fontSize=20, + anchor="start", + color="black", + ) + + return fig + + +def compare_work_district( + tablesets, + district_id, + label="district", + hometaz_col="home_zone_id", + worktaz_col="workplace_zone_id", + data_dictionary=None, +): + data_dictionary = check_data_dictionary(data_dictionary) + + d = {} + h = f"home_{label}" + w = f"work_{label}" + + for key, tableset in tablesets.items(): + persons = tableset["persons"] + workers = persons[persons[worktaz_col] >= 0].copy() + district_map = tableset["land_use"][district_id] + # workers[f"home_{label}_"] = workers[hometaz_col].map(district_map) + # workers[f"work_{label}_"] = workers[worktaz_col].map(district_map) + home_district = workers[hometaz_col].map(district_map).rename(h) + work_district = workers[worktaz_col].map(district_map).rename(w) + df = ( + workers.groupby( + [home_district, work_district] + # [f"home_{label}_", f"work_{label}_"] + ) + .size() + .rename("n_workers") + ) + d[key] = df + + all_d = pd.concat(d, names=["source"]).reset_index() + + district_names = data_dictionary.get("land_use", {}).get(district_id, None) + if district_names is not None: + all_d[h] = all_d[h].map(district_names) + all_d[w] = all_d[w].map(district_names) + + selection = alt.selection_multi( + fields=[w], + bind="legend", + ) + + fig = ( + alt.Chart(all_d) + .mark_bar() + .encode( + color=f"{w}:N", + y=alt.Y("source", axis=alt.Axis(grid=False, title=""), sort=None), + x=alt.X("n_workers", axis=alt.Axis(grid=False)), + row=f"{h}:N", + opacity=alt.condition(selection, alt.value(1), alt.value(0.2)), + tooltip=[f"{h}:N", f"{w}:N", "source", "n_workers"], + ) + .add_selection( + selection, + ) + ) + + return fig + + +def compare_runtime(combo_timing_log): + df = pd.read_csv(combo_timing_log, index_col="model_name") + df1 = ( + df[["sharrow", "legacy"]] + .rename_axis(columns="source") + .unstack() + .rename("seconds") + .reset_index() + ) + c = alt.Chart( + df1, + height={"step": 20}, + ) + + result = c.mark_bar(yOffset=-3, size=6,).transform_filter( + (alt.datum.source == "legacy") + ).encode( + x=alt.X("seconds:Q", stack=None), + y=alt.Y("model_name", type="nominal", sort=None), + color="source", + tooltip=["source", "model_name", "seconds"], + ) + c.mark_bar( + yOffset=4, + size=6, + ).transform_filter( + (alt.datum.source == "sharrow") + ).encode( + x=alt.X("seconds:Q", stack=None), + y=alt.Y("model_name", type="nominal", sort=None), + color="source", + tooltip=["source", "model_name", "seconds"], + ) | alt.Chart( + df1 + ).mark_bar().encode( + color="source", x="source", y="sum(seconds)", tooltip=["source", "sum(seconds)"] + ) + + return result diff --git a/activitysim/standalone/data_dictionary.py b/activitysim/standalone/data_dictionary.py new file mode 100644 index 0000000000..a4fc9e4b78 --- /dev/null +++ b/activitysim/standalone/data_dictionary.py @@ -0,0 +1,34 @@ +import os.path +import warnings +from typing import Mapping + +import yaml + + +def check_data_dictionary(input): + """ + Read and validate a data dictionary. + + The dictionary should be a nested mapping, with top level keys + giving table names, second level keys giving column names, and + then finally a mapping of codes to values. + """ + if input is None: + return {} + elif isinstance(input, str): + if not os.path.exists(input): + warnings.warn(f"data dictionary file {input} is missing") + return {} + with open(input, "rt") as f: + content = yaml.safe_load(f) + else: + content = input + + for i, v in content.items(): + assert isinstance(i, str) + assert isinstance(v, Mapping) + for c, j in v.items(): + assert isinstance(c, str) + assert isinstance(j, Mapping) + + return content diff --git a/activitysim/standalone/pipeline.py b/activitysim/standalone/pipeline.py new file mode 100644 index 0000000000..948914d102 --- /dev/null +++ b/activitysim/standalone/pipeline.py @@ -0,0 +1,45 @@ +import pandas as pd + +from ..core import pipeline + + +def load_checkpointed_tables( + pipeline_file_path, + tables=None, + checkpoint_name=None, +): + pipeline_store = pd.HDFStore(pipeline_file_path, mode="r") + + checkpoints = pipeline_store[pipeline.CHECKPOINT_TABLE_NAME] + + # checkpoint row as series + if checkpoint_name is None: + checkpoint = checkpoints.iloc[-1] + checkpoint_name = checkpoint.loc[pipeline.CHECKPOINT_NAME] + else: + i = checkpoints.set_index(pipeline.CHECKPOINT_NAME).index.get_loc( + checkpoint_name + ) + checkpoint = checkpoints.iloc[i] + + # series with table name as index and checkpoint_name as value + checkpoint_tables = checkpoint[~checkpoint.index.isin(pipeline.NON_TABLE_COLUMNS)] + + # omit dropped tables with empty checkpoint name + checkpoint_tables = checkpoint_tables[checkpoint_tables != ""] + + # hdf5 key is / + checkpoint_tables = { + table_name: pipeline.pipeline_table_key(table_name, checkpoint_name) + for table_name, checkpoint_name in checkpoint_tables.items() + } + + data = {} + for table_name, table_key in checkpoint_tables.items(): + if tables is None or table_name in tables: + data[table_name] = pipeline_store[table_key] + + pipeline_store.close() + + # checkpoint name and series mapping table name to hdf5 key for tables in that checkpoint + return checkpoint_name, data diff --git a/activitysim/standalone/render.py b/activitysim/standalone/render.py new file mode 100644 index 0000000000..ad672a7d19 --- /dev/null +++ b/activitysim/standalone/render.py @@ -0,0 +1,64 @@ +import logging +import os +import textwrap +from contextlib import contextmanager +from pathlib import Path + +import nbclient +import nbformat as nbf +from nbconvert import HTMLExporter +from xmle import NumberedCaption, Reporter + +from .. import __version__ + +# from jupyter_contrib_nbextensions.nbconvert_support import TocExporter # problematic + + +@contextmanager +def chdir(path: Path): + """ + Sets the cwd within the context + + Args: + path (Path): The path to the cwd + + Yields: + None + """ + + cwd = Path().absolute() + try: + os.chdir(path) + yield + finally: + os.chdir(cwd) + + +def render_notebook(nb_filename, cellcontent): + nb_filename = os.path.splitext(nb_filename)[0] + nb = nbf.v4.new_notebook() + + cells = [] + for c in cellcontent: + c = textwrap.dedent(c).strip() + if c[:4] == "[md]": + cells.append(nbf.v4.new_markdown_cell(c[4:])) + else: + cells.append(nbf.v4.new_code_cell(c)) + nb["cells"] = cells + nbf.write(nb, nb_filename + ".ipynb") + + nb = nbclient.execute(nb, cwd=os.path.dirname(nb_filename)) + nbf.write(nb, nb_filename + "-e.ipynb") + + html_exporter = HTMLExporter( + embed_images=True, + exclude_input_prompt=True, + exclude_output_prompt=True, + exclude_input=True, + # template_name = 'classic' + ) + (body, resources) = html_exporter.from_notebook_node(nb) + + with open(nb_filename + ".html", "w") as f: + f.write(body) diff --git a/activitysim/standalone/skims.py b/activitysim/standalone/skims.py new file mode 100644 index 0000000000..1065383c5d --- /dev/null +++ b/activitysim/standalone/skims.py @@ -0,0 +1,64 @@ +import glob +import logging +import os + +import numpy as np +import openmatrix +import sharrow as sh +import yaml + +logger = logging.getLogger(__name__) + + +def load_skims( + network_los_settings_filename, + data_dir, +): + with open(network_los_settings_filename, "rt") as f: + settings = yaml.safe_load(f) + + skim_settings = settings["taz_skims"] + if isinstance(skim_settings, str): + skims_omx_fileglob = skim_settings + else: + skims_omx_fileglob = skim_settings.get("omx", None) + skims_omx_fileglob = skim_settings.get("files", skims_omx_fileglob) + skims_filenames = glob.glob(os.path.join(data_dir, skims_omx_fileglob)) + index_names = ("otaz", "dtaz", "time_period") + indexes = None + time_period_breaks = settings.get("skim_time_periods", {}).get("periods") + time_periods = settings.get("skim_time_periods", {}).get("labels") + time_period_sep = "__" + + time_window = settings.get("skim_time_periods", {}).get("time_window") + period_minutes = settings.get("skim_time_periods", {}).get("period_minutes") + n_periods = int(time_window / period_minutes) + + tp_map = {} + tp_imap = {} + label = time_periods[0] + i = 0 + for t in range(n_periods): + if t in time_period_breaks: + i = time_period_breaks.index(t) + label = time_periods[i] + tp_map[t + 1] = label + tp_imap[t + 1] = i + + omxs = [ + openmatrix.open_file(skims_filename, mode="r") + for skims_filename in skims_filenames + ] + if isinstance(time_periods, (list, tuple)): + time_periods = np.asarray(time_periods) + result = sh.dataset.from_omx_3d( + omxs, + index_names=index_names, + indexes=indexes, + time_periods=time_periods, + time_period_sep=time_period_sep, + ) + result.attrs["time_period_map"] = tp_map + result.attrs["time_period_imap"] = tp_imap + + return result diff --git a/activitysim/standalone/utils.py b/activitysim/standalone/utils.py new file mode 100644 index 0000000000..7ba7c4ca4d --- /dev/null +++ b/activitysim/standalone/utils.py @@ -0,0 +1,24 @@ +import os +from contextlib import contextmanager +from pathlib import Path + + +@contextmanager +def chdir(path: Path): + """ + Sets the cwd within the context + + Args: + path (Path): The path to the cwd + + Yields: + None + """ + + cwd = Path().absolute() + try: + if path is not None: + os.chdir(path) + yield + finally: + os.chdir(cwd) diff --git a/activitysim/workflows/__init__.py b/activitysim/workflows/__init__.py new file mode 100644 index 0000000000..35ea4784b9 --- /dev/null +++ b/activitysim/workflows/__init__.py @@ -0,0 +1,7 @@ +try: + import pypyr +except ImportError: + pass +else: + from .steps import cmd, py + from .steps.main import get_pipeline_definition, main diff --git a/activitysim/workflows/demo.yaml b/activitysim/workflows/demo.yaml new file mode 100644 index 0000000000..750ddd997a --- /dev/null +++ b/activitysim/workflows/demo.yaml @@ -0,0 +1,39 @@ +context_parser: pypyr.parser.keyvaluepairs +steps: + +- name: pypyr.steps.default + in: + defaults: + example_name: prototype_mtc + workspace: workspace + legacy: True + tag: + stress: 22 + +- name: activitysim.workflows.steps.title + in: + label: "[bold blue]activitysim workflow demo {example_name}" + +- description: Make {workspace} directory if it does not exist + name: activitysim.workflows.steps.py + in: + py: | + import os + os.makedirs(f"{workspace}", exist_ok=True) + +- description: Generate a tag based on datetime if tag is not given + name: activitysim.workflows.steps.py + in: + label: Generate tag + py: | + import time + if tag is None: + tag = time.strftime("%Y-%m-%d-%H%M%S") + time.sleep(2) + save(tag=tag) + print(f"tag is {tag}") + +- description: Memory Stress Test + name: activitysim.workflows.steps.memory_stress_test + in: + n: '{stress}' diff --git a/activitysim/workflows/sharrow-contrast/_contrast_runner.yaml b/activitysim/workflows/sharrow-contrast/_contrast_runner.yaml new file mode 100644 index 0000000000..65788bf8f7 --- /dev/null +++ b/activitysim/workflows/sharrow-contrast/_contrast_runner.yaml @@ -0,0 +1,688 @@ +# activitysim workflow example_runner example_name=prototype_mtc + +context_parser: pypyr.parser.keyvaluepairs + +on_failure: +- name: activitysim.workflows.steps.py + in: + py: | + import time + print("FAILURE", time.strftime("%Y-%m-%d %I:%M:%S%p")) + +steps: + +- description: Setting default workflow parameters + name: pypyr.steps.default + in: + defaults: + example_name: prototype_mtc + workflow_name: sharrow-contrast + workspace: workspace + multiprocess: 0 + chunk_size: 0 + create: True + compile: True + sharrow: True + legacy: True + reference: False + reference_asim_version: "1.0.4" + tag: + resume_after: + fast: False + compile_n_households: 1000 + main_n_households: 100000 + config_dirs: configs + data_dir: data + ext_dirs: + instrument: False + memory_profile: False + trace_hh_id: + trace_od: + chunk_method: hybrid_uss + chunk_training_mode: disabled + machine_name: + disable_zarr: False + settings_file: settings.yaml + report_skip: [] + +- name: activitysim.workflows.steps.title + in: + label: "activitysim workflow {workflow_name}" + formatting: bold cyan + +- name: activitysim.workflows.steps.py + in: + label: Make {workspace} directory if it does not exist + py: | + import os + os.makedirs(f"{workspace}", exist_ok=True) + +- name: activitysim.workflows.steps.py + in: + label: Detect if debugging + py: | + def is_debug(): + import sys + gettrace = getattr(sys, 'gettrace', None) + if gettrace is None: + return False + else: + v = gettrace() + if v is None: + return False + else: + return True + should_swallow_errors = not is_debug() + save('should_swallow_errors') + +- activitysim.workflows.steps.contrast.contrast_setup + +- name: activitysim.workflows.steps.create + run: '{create}' + in: + destination: "{workspace}" + +- activitysim.workflows.steps.contrast.directory_prep + +- name: pypyr.steps.call + run: '{compile}' + in: + call: run-compile + swallow: False + +- name: pypyr.steps.call + run: '{sharrow}' + in: + call: run-sharrow + swallow: '{should_swallow_errors}' + +- name: pypyr.steps.call + run: '{legacy}' + in: + call: run-legacy + swallow: '{should_swallow_errors}' + +- name: pypyr.steps.call + run: '{reference}' + in: + call: run-reference + swallow: '{should_swallow_errors}' + +- activitysim.workflows.steps.contrast.composite_log + +- name: pypyr.steps.call + in: + call: + groups: reporting + success: report-save + failure: report-save + swallow: False + +################################################################################ +run-compile: + # This step group runs activitysim with a (usually) smaller sample of + # household, to generate the compiled numba code for the local machine + +- description: write configs_sh_compile + name: pypyr.steps.filewriteyaml + in: + fileWriteYaml: + path: "{workspace}/{example_name}/configs_sh_compile/{settings_file}" + payload: + inherit_settings: True + sharrow: test + chunk_training_mode: disabled + households_sample_size: '{compile_n_households}' + cache_dir: cache_sharrow + trace_hh_id: '{trace_hh_id}' + trace_od: '{trace_od}' + instrument: '{instrument}' + disable_zarr: '{disable_zarr}' + multiprocess: False + +- description: Run activitysim to compile and test sharrow-enabled model + name: activitysim.workflows.steps.run + in: + pre_config_dirs: configs_sh_compile + output_dir: 'output-{tag}/output-compile' + cwd: "{workspace}/{example_name}" + label: "{example_name} -- sharrow compile" + + +################################################################################ +run-sharrow: + +- description: write configs_sh + name: pypyr.steps.filewriteyaml + in: + fileWriteYaml: + path: "{workspace}/{example_name}/configs_sh/{settings_file}" + payload: + inherit_settings: True + sharrow: require + cache_dir: cache_sharrow + households_sample_size: '{main_n_households}' + trace_hh_id: '{trace_hh_id}' + trace_od: '{trace_od}' + instrument: '{instrument}' + memory_profile: '{memory_profile}' + chunk_size: '{chunk_size}' + num_processes: '{num_processes}' + multiprocess: '{is_multiprocess}' + chunk_method: '{chunk_method}' + chunk_training_mode: '{chunk_training_mode}' + disable_zarr: '{disable_zarr}' + +- description: Run activitysim to evaluate sharrow-enabled model + name: activitysim.workflows.steps.run_subprocess + in: + pre_config_dirs: configs_sh + output_dir: 'output-{tag}/output-sharrow' + cwd: "{workspace}/{example_name}" + label: "{example_name} -- sharrow run" + +################################################################################ +run-legacy: + +- description: write.configs_legacy + name: pypyr.steps.filewriteyaml + in: + fileWriteYaml: + path: "{workspace}/{example_name}/configs_legacy/{settings_file}" + payload: + inherit_settings: True + recode_pipeline_columns: False + cache_dir: cache_legacy + households_sample_size: '{main_n_households}' + trace_hh_id: '{trace_hh_id}' + trace_od: '{trace_od}' + instrument: '{instrument}' + memory_profile: '{memory_profile}' + chunk_size: '{chunk_size}' + num_processes: '{num_processes}' + multiprocess: '{is_multiprocess}' + chunk_method: '{chunk_method}' + chunk_training_mode: '{chunk_training_mode}' + +- description: Run activitysim to evaluate legacy model + name: activitysim.workflows.steps.run_subprocess + in: + pre_config_dirs: configs_legacy + output_dir: 'output-{tag}/output-legacy' + cwd: "{workspace}/{example_name}" + label: "{example_name} -- legacy run" + + +################################################################################ +run-reference: + +- description: Install a reference environment + name: activitysim.workflows.steps.install_env + in: + label: "{example_name} -- install ref env {reference_asim_version}" + env_prefix: "{workspace}/env/asim-ref-{reference_asim_version}" + asim_version: '{reference_asim_version}' + +- name: activitysim.workflows.steps.py + in: + label: Identify first config dir + py: | + if isinstance(config_dirs, str): + first_config_dir = config_dirs + else: + first_config_dir = config_dirs[0] + save('first_config_dir') + +- description: Copy required reference settings + name: activitysim.workflows.steps.copy_files + in: + source_glob: + - "{workspace}/{example_name}/{first_config_dir}/legacy-{reference_asim_version}/*.yaml" + - "{workspace}/{example_name}/{first_config_dir}/legacy-{reference_asim_version}/*.csv" + dest_dir: "{workspace}/{example_name}/configs_reference" + +- name: activitysim.workflows.steps.py + in: + label: Allow resume_after in ref only if pipeline exists + # Older versions of ActivitySim choke on resume_after if pipeline is missing. + py: | + import os + ref_pipeline = f"{workspace}/{example_name}/output-{tag}/output-reference/pipeline.h5" + if os.path.exists(ref_pipeline): + resume_after_ref = resume_after + else: + resume_after_ref = None + save('resume_after_ref') + +- description: write.configs_reference + name: activitysim.workflows.steps.update_yaml + in: + updateYaml: + path: "{workspace}/{example_name}/configs_reference/{settings_file}" + payload: + inherit_settings: True + households_sample_size: '{main_n_households}' + trace_hh_id: '{trace_hh_id}' + trace_od: '{trace_od}' + resume_after: '{resume_after_ref}' + chunk_size: '{chunk_size}' + num_processes: '{num_processes}' + multiprocess: '{is_multiprocess}' + chunk_method: '{chunk_method}' + chunk_training_mode: '{chunk_training_mode}' + +- description: Run activitysim to evaluate reference model + name: activitysim.workflows.steps.run_subprocess + in: + resume_after: + pre_config_dirs: configs_reference + output_dir: 'output-{tag}/output-reference' + cwd: "{workspace}/{example_name}" + label: "{example_name} -- reference run" + conda_prefix: "../env/asim-ref-{reference_asim_version}" + + +################################################################################ +reporting: + +- name: activitysim.workflows.steps.contrast.load_tables + in: + common_output_directory: "{workspace}/{example_name}/output-{tag}" + databases: + sharrow: "output-sharrow" + legacy: "output-legacy" + reference: "output-reference" + tables: + households: + filename: final_households.csv + index_col: household_id + persons: + filename: final_persons.csv + index_col: person_id + tours: + filename: final_tours.csv + index_col: tour_id + trips: + filename: final_trips.csv + index_col: trip_id + land_use: + filename: final_land_use.csv + index_col: zone_id + +- name: activitysim.workflows.steps.contrast.load_skims + in: + common_directory: "{workspace}/{example_name}" + +- name: activitysim.workflows.steps.reporting.init_report + in: + title: "{example_name} report" + common_directory: "{workspace}/{example_name}" + +- name: activitysim.workflows.steps.reporting.machine_info + in: + caption: Machine Info + +- name: activitysim.workflows.steps.reporting.settings + in: + caption: Settings + names: + - disable_zarr + - resume_after + - instrument + - memory_profile + - fast + - chunk_method + - chunk_training_mode + - chunk_size + - multiprocess + + +#### Runtime and Data Inventory #### + +- name: activitysim.workflows.steps.contrast.runtime + in: + caption: Model Runtime + include_runs: + - reference + - legacy + - sharrow + +- name: activitysim.workflows.steps.reporting.section_title + in: + title: Memory Usage + +- name: activitysim.workflows.steps.contrast.memory_use + in: + caption: USS + caption_level: 3 + memory_measure: uss + include_runs: + - reference + - legacy + - sharrow + +- name: activitysim.workflows.steps.contrast.memory_use + in: + caption: RSS + caption_level: 3 + memory_measure: rss + include_runs: + - reference + - legacy + - sharrow + +- name: activitysim.workflows.steps.contrast.memory_use + in: + caption: Full RSS + caption_level: 3 + memory_measure: full_rss + include_runs: + - reference + - legacy + - sharrow + +- name: activitysim.workflows.steps.contrast.memory_use_peak + in: + caption: Peak RSS by Component + caption_level: 3 + memory_measure: rss + include_runs: + - reference + - legacy + - sharrow + +- name: activitysim.workflows.steps.contrast.memory_use_peak + in: + caption: Peak USS by Component + caption_level: 3 + memory_measure: uss + include_runs: + - reference + - legacy + - sharrow + + +- name: activitysim.workflows.steps.contrast.data_inventory + + +#### Workplace Location #### + +- name: activitysim.workflows.steps.reporting.load_data_dictionary + in: + cwd: "{workspace}/{example_name}" + +- name: pypyr.steps.default + in: + defaults: + workplace_zone_agg: + +- name: activitysim.workflows.steps.reporting.section_title + skip: !py "'work location' in report_skip" + run: '{workplace_zone_agg}' + in: + title: Workplace Location + +- name: activitysim.workflows.steps.contrast.district_to_district + skip: !py "'work location' in report_skip" + run: '{workplace_zone_agg}' + in: + tablename: persons + caption: '{workplace_zone_agg[caption]}' + caption_level: 3 + district_id: '{workplace_zone_agg[district_id]}' + orig_label: home district + dest_label: work district + orig_col: home_zone_id + dest_col: workplace_zone_id + filter: workplace_zone_id >= 0 + size_label: n_workers + +- name: activitysim.workflows.steps.contrast.transform_data + in: + tablename: households + column: '{household_income}' + out: income_tertile + qcut: + q: 3 + labels: + - low + - mid + - high + +- name: activitysim.workflows.steps.contrast.join_table_data + in: + caption: Join Income to Persons + tablename: persons + from_tablename: households + columns: income_tertile + on: household_id + +- name: activitysim.workflows.steps.contrast.trip_distance + skip: !py "'work location' in report_skip" + in: + caption: Workplace Distance by Income + caption_level: 3 + dist_bins: 20 + dist_skim_name: 'distance_to_work' + tablename: persons + grouping: income_tertile + +- name: activitysim.workflows.steps.contrast.trip_distance + skip: !py "'school location' in report_skip" + in: + caption: School Distance by Income + caption_level: 3 + dist_bins: 20 + dist_skim_name: 'distance_to_school' + tablename: persons + grouping: income_tertile + +#### Auto Ownership #### + +- name: activitysim.workflows.steps.reporting.section_title + skip: !py "'auto ownership' in report_skip" + in: + title: Auto Ownership + +- name: activitysim.workflows.steps.contrast.transform_data + skip: !py "'auto ownership' in report_skip" + in: + tablename: households + column: hhsize + out: hhsize_to5 + clip: + upper: 5 + +- name: activitysim.workflows.steps.contrast.nominal_choice + skip: !py "'auto ownership' in report_skip" + in: + caption: Household Auto Ownership Counts by Household Size + caption_level: 3 + tablename: households + nominal_col: auto_ownership + row_grouping: + field: hhsize_to5 + title: Household Size (up to 5) + col_grouping: + field: income_tertile + sort: [low, mid, high] + title: Income Tertile + ordinal: true + plot_type: count + axis_label: "# of Households" + +- name: activitysim.workflows.steps.contrast.nominal_choice + skip: !py "'auto ownership' in report_skip" + in: + caption: Household Auto Ownership Shares by Household Size + caption_level: 3 + tablename: households + nominal_col: auto_ownership + row_grouping: + field: hhsize_to5 + title: Household Size (up to 5) + col_grouping: + field: income_tertile + sort: [low, mid, high] + title: Income Tertile + ordinal: true + + +#### CDAP #### + +- name: activitysim.workflows.steps.reporting.section_title + skip: !py "'cdap' in report_skip" + in: + title: Coordinated Daily Activity Pattern + +- name: activitysim.workflows.steps.contrast.nominal_choice + skip: !py "'cdap' in report_skip" + in: + caption: Coordinated Daily Activity Pattern by Person Type + caption_level: 3 + tablename: persons + nominal_col: cdap_activity + row_grouping: + field: ptype + title: Person Type + axis_label: Daily Activity Pattern Count + plot_type: count + + +#### Tour Mode Choice #### + +- name: activitysim.workflows.steps.reporting.section_title + in: + title: Tour Mode Choice + +- name: activitysim.workflows.steps.contrast.nominal_choice + in: + caption: Tour Mode Choice by Primary Purpose + caption_level: 3 + tablename: tours + nominal_col: tour_mode + row_grouping: primary_purpose + axis_label: Tour Mode Share + +- name: activitysim.workflows.steps.contrast.nominal_choice + skip: !py "'tour mode by composition' in report_skip" + in: + caption: Tour Mode Choice by Composition + caption_level: 3 + tablename: tours + nominal_col: tour_mode + row_grouping: composition + axis_label: Tour Mode Share + + +#### Tour Schedule #### + +- name: activitysim.workflows.steps.reporting.section_title + in: + title: Tour Scheduling + +- name: activitysim.workflows.steps.contrast.ordinal_distribution + in: + caption: Tour Start Time by Primary Purpose + caption_level: 3 + tablename: tours + ordinal_col: start + facet_grouping: primary_purpose + plot_type: count + +- name: activitysim.workflows.steps.contrast.ordinal_distribution + in: + caption: Tour Duration by Primary Purpose + caption_level: 3 + tablename: tours + ordinal_col: duration + facet_grouping: primary_purpose + plot_type: count + + +#### Trip Mode Choice #### + +- name: activitysim.workflows.steps.reporting.section_title + in: + title: Trip Mode Choice + +- name: activitysim.workflows.steps.contrast.nominal_choice + in: + caption: Trip Mode Choice by Primary Purpose + caption_level: 3 + tablename: trips + nominal_col: trip_mode + row_grouping: primary_purpose + axis_label: Trip Mode Share + +- name: activitysim.workflows.steps.contrast.nominal_choice + in: + caption: Trip Mode Choice by Departure Time + caption_level: 3 + tablename: trips + nominal_col: trip_mode + row_grouping: depart + axis_label: Trip Mode Share + + +#### Trip Distance #### + +- name: activitysim.workflows.steps.reporting.section_title + in: + title: Trip Distance + +- name: activitysim.workflows.steps.contrast.attach_skim_data + in: + tablename: trips + otaz_col: origin + dtaz_col: destination + time_col: depart + skim_vars: '{distance_skim}' + +- name: activitysim.workflows.steps.contrast.transform_data + in: + tablename: trips + column: '{distance_skim}' + out: distance_to10_binned + censor: + left: 0 + right: 10 + cut: + bins: 20 + labels: midpoint + +- name: activitysim.workflows.steps.contrast.ordinal_distribution + in: + caption: Trip Distance by the Primary Purpose, <10 miles + caption_level: 3 + tablename: trips + ordinal_col: distance_to10_binned + facet_grouping: primary_purpose + plot_type: count + interpolate: step + value_format: "0.2f" + axis_label: Distance (to 10 miles) + +- name: activitysim.workflows.steps.contrast.trip_distance + in: + caption: Trip Distance by Primary Purpose, <10 miles + caption_level: 3 + grouping: primary_purpose + dist_bins: 20 + dist_skim_name: '{distance_skim}' + max_dist: 10 + +- name: activitysim.workflows.steps.contrast.trip_distance + in: + caption: Trip Distance by Primary Purpose + caption_level: 3 + grouping: primary_purpose + dist_bins: 20 + dist_skim_name: '{distance_skim}' + + +################################################################################ +report-save: +- name: activitysim.workflows.steps.reporting.save_report + in: + html_filename: "{workspace}/{example_name}/output-{tag}/report-{tag}.html" diff --git a/activitysim/workflows/sharrow-contrast/_contrast_runner_chunked.yaml b/activitysim/workflows/sharrow-contrast/_contrast_runner_chunked.yaml new file mode 100644 index 0000000000..f46bbca571 --- /dev/null +++ b/activitysim/workflows/sharrow-contrast/_contrast_runner_chunked.yaml @@ -0,0 +1,668 @@ +# activitysim workflow example_runner example_name=prototype_mtc + +context_parser: pypyr.parser.keyvaluepairs + +on_failure: +- name: activitysim.workflows.steps.py + in: + py: | + import time + print("FAILURE", time.strftime("%Y-%m-%d %I:%M:%S%p")) + +steps: + +- description: Setting default workflow parameters + name: pypyr.steps.default + in: + defaults: + example_name: prototype_mtc + workflow_name: sharrow-contrast + workspace: workspace + multiprocess: 0 + chunk_size: + chunk_size_pct_of_available: 0.75 + create: True + compile: True + sharrow: True + legacy: False + reference: False + reference_asim_version: "1.0.4" + tag: + resume_after: + fast: False + compile_n_households: 1000 + train_n_households: 20000 + main_n_households: 100000 + config_dirs: configs + data_dir: data + instrument: False + memory_profile: False + trace_hh_id: + trace_od: + chunk_method: hybrid_uss + chunk_training: True + chunk_training_mode: training + machine_name: + disable_zarr: False + settings_file: settings.yaml + +- name: activitysim.workflows.steps.title + in: + label: "activitysim workflow {workflow_name}" + formatting: bold cyan + +- name: activitysim.workflows.steps.py + in: + label: Make {workspace} directory if it does not exist + py: | + import os + os.makedirs(f"{workspace}", exist_ok=True) + +- activitysim.workflows.steps.contrast.contrast_setup + +- activitysim.workflows.steps.chunk_sizing + +- name: activitysim.workflows.steps.create + run: '{create}' + in: + destination: "{workspace}" + +- activitysim.workflows.steps.contrast.directory_prep + +- name: pypyr.steps.call + run: '{compile}' + in: + call: run-compile + swallow: True + +- name: pypyr.steps.call + run: '{sharrow}' + in: + call: run-sharrow + swallow: True + +- name: pypyr.steps.call + run: '{legacy}' + in: + call: run-legacy + swallow: True + +- name: pypyr.steps.call + run: '{reference}' + in: + call: run-reference + swallow: True + +- activitysim.workflows.steps.contrast.composite_log + +- name: pypyr.steps.call + in: + call: + groups: reporting + success: report-save + failure: report-save + +################################################################################ +run-compile: + # This step group runs activitysim with a (usually) smaller sample of + # household, to generate the compiled numba code for the local machine + +- description: write configs_sh_compile + name: pypyr.steps.filewriteyaml + in: + fileWriteYaml: + path: "{workspace}/{example_name}/configs_sh_compile/{settings_file}" + payload: + inherit_settings: True + sharrow: test + chunk_training_mode: disabled + households_sample_size: '{compile_n_households}' + cache_dir: cache_sharrow + trace_hh_id: + trace_od: + instrument: '{instrument}' + disable_zarr: '{disable_zarr}' + multiprocess: False + +- description: Run activitysim to compile and test sharrow-enabled model + name: activitysim.workflows.steps.run + in: + pre_config_dirs: configs_sh_compile + output_dir: 'output-{tag}/output-compile' + cwd: "{workspace}/{example_name}" + label: "{example_name} -- sharrow compile" + + +################################################################################ +run-sharrow: + +- description: write configs_sh_chunktrain + name: pypyr.steps.filewriteyaml + run: '{chunk_training}' + in: + fileWriteYaml: + path: "{workspace}/{example_name}/configs_sh_chunktrain/{settings_file}" + payload: + inherit_settings: True + chunk_training_mode: '{chunk_training_mode}' + households_sample_size: '{train_n_households}' + num_processes: 2 + +- description: write configs_sh + name: pypyr.steps.filewriteyaml + in: + fileWriteYaml: + path: "{workspace}/{example_name}/configs_sh/{settings_file}" + payload: + inherit_settings: True + sharrow: require + cache_dir: cache_sharrow + households_sample_size: '{main_n_households}' + trace_hh_id: + trace_od: + instrument: '{instrument}' + memory_profile: '{memory_profile}' + chunk_size: '{chunk_size}' + num_processes: '{num_processes}' + multiprocess: '{is_multiprocess}' + chunk_method: '{chunk_method}' + chunk_training_mode: '{chunk_application_mode}' + disable_zarr: '{disable_zarr}' + +- description: Run activitysim to chunk-train sharrow-enabled model + name: activitysim.workflows.steps.run_subprocess + run: '{chunk_training}' + in: + pre_config_dirs: [configs_sh_chunktrain, configs_sh] + output_dir: 'output-{tag}/output-sharrow-training' + cwd: "{workspace}/{example_name}" + label: "{example_name} -- sharrow chunk training run" + +- description: Run activitysim to evaluate sharrow-enabled model + name: activitysim.workflows.steps.run_subprocess + in: + pre_config_dirs: configs_sh + output_dir: 'output-{tag}/output-sharrow' + cwd: "{workspace}/{example_name}" + label: "{example_name} -- sharrow run" + +################################################################################ +run-legacy: + +- description: write configs_legacy_chunktrain + name: pypyr.steps.filewriteyaml + run: '{chunk_training}' + in: + fileWriteYaml: + path: "{workspace}/{example_name}/configs_legacy_chunktrain/{settings_file}" + payload: + inherit_settings: True + chunk_training_mode: '{chunk_training_mode}' + households_sample_size: '{train_n_households}' + num_processes: 2 + +- description: write.configs_legacy + name: pypyr.steps.filewriteyaml + in: + fileWriteYaml: + path: "{workspace}/{example_name}/configs_legacy/{settings_file}" + payload: + inherit_settings: True + recode_pipeline_columns: False + cache_dir: cache_legacy + households_sample_size: '{main_n_households}' + trace_hh_id: '{trace_hh_id}' + trace_od: '{trace_od}' + instrument: '{instrument}' + memory_profile: '{memory_profile}' + chunk_size: '{chunk_size}' + num_processes: '{num_processes}' + multiprocess: '{is_multiprocess}' + chunk_method: '{chunk_method}' + chunk_training_mode: '{chunk_application_mode}' + +- description: Run activitysim to chunk-train legacy model + name: activitysim.workflows.steps.run_subprocess + run: '{chunk_training}' + in: + pre_config_dirs: [configs_legacy_chunktrain, configs_legacy] + output_dir: 'output-{tag}/output-legacy-training' + cwd: "{workspace}/{example_name}" + label: "{example_name} -- legacy chunk training run" + +- description: Run activitysim to evaluate legacy model + name: activitysim.workflows.steps.run_subprocess + in: + pre_config_dirs: configs_legacy + output_dir: 'output-{tag}/output-legacy' + cwd: "{workspace}/{example_name}" + label: "{example_name} -- legacy run" + + +################################################################################ +run-reference: + +- description: Install a reference environment + name: activitysim.workflows.steps.install_env + in: + label: "{example_name} -- install ref env {reference_asim_version}" + env_prefix: "{workspace}/env/asim-ref-{reference_asim_version}" + asim_version: '{reference_asim_version}' + +- description: Copy required reference settings + name: activitysim.workflows.steps.copy_files + in: + source_glob: + - "{workspace}/{example_name}/configs/legacy-{reference_asim_version}/*.yaml" + - "{workspace}/{example_name}/configs/legacy-{reference_asim_version}/*.csv" + dest_dir: "{workspace}/{example_name}/configs_reference" + +- name: activitysim.workflows.steps.py + in: + label: Allow resume_after in ref only if pipeline exists + # Older versions of ActivitySim choke on resume_after if pipeline is missing. + py: | + import os + ref_pipeline = f"{workspace}/{example_name}/output-{tag}/output-reference/pipeline.h5" + if os.path.exists(ref_pipeline): + resume_after_ref = resume_after + else: + resume_after_ref = None + save('resume_after_ref') + +- description: write.configs_reference + name: activitysim.workflows.steps.update_yaml + in: + updateYaml: + path: "{workspace}/{example_name}/configs_reference/settings.yaml" + payload: + inherit_settings: True + households_sample_size: '{main_n_households}' + trace_hh_id: '{trace_hh_id}' + trace_od: '{trace_od}' + resume_after: '{resume_after_ref}' + chunk_size: '{chunk_size}' + num_processes: '{num_processes}' + multiprocess: '{is_multiprocess}' + chunk_method: '{chunk_method}' + chunk_training_mode: '{chunk_application_mode}' + +- description: Run activitysim to evaluate reference model + name: activitysim.workflows.steps.run_subprocess + in: + resume_after: + pre_config_dirs: configs_reference + output_dir: 'output-{tag}/output-reference' + cwd: "{workspace}/{example_name}" + label: "{example_name} -- reference run" + conda_prefix: "../env/asim-ref-{reference_asim_version}" + + +################################################################################ +reporting: + +- name: activitysim.workflows.steps.contrast.load_tables + in: + common_output_directory: "{workspace}/{example_name}/output-{tag}" + databases: + sharrow: "output-sharrow" + legacy: "output-legacy" + reference: "output-reference" + tables: + households: + filename: final_households.csv + index_col: household_id + persons: + filename: final_persons.csv + index_col: person_id + tours: + filename: final_tours.csv + index_col: tour_id + trips: + filename: final_trips.csv + index_col: trip_id + land_use: + filename: final_land_use.csv + index_col: zone_id + +- name: activitysim.workflows.steps.contrast.load_skims + in: + common_directory: "{workspace}/{example_name}" + +- name: activitysim.workflows.steps.reporting.init_report + in: + title: "{example_name} report" + common_directory: "{workspace}/{example_name}" + +- name: activitysim.workflows.steps.reporting.machine_info + in: + caption: Machine Info + +- name: activitysim.workflows.steps.reporting.settings + in: + caption: Settings + names: + - disable_zarr + - resume_after + - instrument + - memory_profile + - fast + - chunk_method + - chunk_training_mode + - chunk_size + - multiprocess + + +#### Runtime and Data Inventory #### + +- name: activitysim.workflows.steps.contrast.runtime + in: + caption: Model Runtime + include_runs: + - reference + - legacy + - sharrow + +- name: activitysim.workflows.steps.reporting.section_title + in: + title: Memory Usage + +- name: activitysim.workflows.steps.contrast.memory_use + in: + caption: USS + caption_level: 3 + memory_measure: uss + include_runs: + - reference + - legacy + - sharrow + +- name: activitysim.workflows.steps.contrast.memory_use + in: + caption: RSS + caption_level: 3 + memory_measure: rss + include_runs: + - reference + - legacy + - sharrow + +- name: activitysim.workflows.steps.contrast.memory_use + in: + caption: Full RSS + caption_level: 3 + memory_measure: full_rss + include_runs: + - reference + - legacy + - sharrow + +- name: activitysim.workflows.steps.contrast.memory_use_peak + in: + caption: Peak RSS by Component + caption_level: 3 + memory_measure: rss + include_runs: + - reference + - legacy + - sharrow + +- name: activitysim.workflows.steps.contrast.memory_use_peak + in: + caption: Peak USS by Component + caption_level: 3 + memory_measure: uss + include_runs: + - reference + - legacy + - sharrow + + +- name: activitysim.workflows.steps.contrast.data_inventory + + +#### Workplace Location #### + +- name: activitysim.workflows.steps.reporting.load_data_dictionary + in: + cwd: "{workspace}/{example_name}" + +- name: pypyr.steps.default + in: + defaults: + workplace_zone_agg: + +- name: activitysim.workflows.steps.reporting.section_title + run: '{workplace_zone_agg}' + in: + title: Workplace Location + +- name: activitysim.workflows.steps.contrast.district_to_district + run: '{workplace_zone_agg}' + in: + tablename: persons + caption: '{workplace_zone_agg[caption]}' + caption_level: 3 + district_id: '{workplace_zone_agg[district_id]}' + orig_label: home district + dest_label: work district + orig_col: home_zone_id + dest_col: workplace_zone_id + filter: workplace_zone_id >= 0 + size_label: n_workers + + +#### Auto Ownership #### + +- name: activitysim.workflows.steps.reporting.section_title + in: + title: Auto Ownership + +- name: activitysim.workflows.steps.contrast.transform_data + in: + tablename: households + column: hhsize + out: hhsize_to5 + clip: + upper: 5 + +- name: activitysim.workflows.steps.contrast.transform_data + in: + tablename: households + column: '{household_income}' + out: income_tertile + qcut: + q: 3 + labels: + - low + - mid + - high + +- name: activitysim.workflows.steps.contrast.nominal_choice + in: + caption: Household Auto Ownership Counts by Household Size + caption_level: 3 + tablename: households + nominal_col: auto_ownership + row_grouping: + field: hhsize_to5 + title: Household Size (up to 5) + col_grouping: + field: income_tertile + sort: [low, mid, high] + title: Income Tertile + ordinal: true + plot_type: count + axis_label: "# of Households" + +- name: activitysim.workflows.steps.contrast.nominal_choice + in: + caption: Household Auto Ownership Shares by Household Size + caption_level: 3 + tablename: households + nominal_col: auto_ownership + row_grouping: + field: hhsize_to5 + title: Household Size (up to 5) + col_grouping: + field: income_tertile + sort: [low, mid, high] + title: Income Tertile + ordinal: true + + +#### CDAP #### + +- name: activitysim.workflows.steps.reporting.section_title + in: + title: Coordinated Daily Activity Pattern + +- name: activitysim.workflows.steps.contrast.nominal_choice + in: + caption: Coordinated Daily Activity Pattern by Person Type + caption_level: 3 + tablename: persons + nominal_col: cdap_activity + row_grouping: + field: ptype + title: Person Type + axis_label: Daily Activity Pattern Count + plot_type: count + + +#### Tour Mode Choice #### + +- name: activitysim.workflows.steps.reporting.section_title + in: + title: Tour Mode Choice + +- name: activitysim.workflows.steps.contrast.nominal_choice + in: + caption: Tour Mode Choice by Primary Purpose + caption_level: 3 + tablename: tours + nominal_col: tour_mode + row_grouping: primary_purpose + axis_label: Tour Mode Share + +- name: activitysim.workflows.steps.contrast.nominal_choice + in: + caption: Tour Mode Choice by Composition + caption_level: 3 + tablename: tours + nominal_col: tour_mode + row_grouping: composition + axis_label: Tour Mode Share + + +#### Tour Schedule #### + +- name: activitysim.workflows.steps.reporting.section_title + in: + title: Tour Scheduling + +- name: activitysim.workflows.steps.contrast.ordinal_distribution + in: + caption: Tour Start Time by Primary Purpose + caption_level: 3 + tablename: tours + ordinal_col: start + facet_grouping: primary_purpose + plot_type: count + +- name: activitysim.workflows.steps.contrast.ordinal_distribution + in: + caption: Tour Duration by Primary Purpose + caption_level: 3 + tablename: tours + ordinal_col: duration + facet_grouping: primary_purpose + plot_type: count + + +#### Trip Mode Choice #### + +- name: activitysim.workflows.steps.reporting.section_title + in: + title: Trip Mode Choice + +- name: activitysim.workflows.steps.contrast.nominal_choice + in: + caption: Trip Mode Choice by Primary Purpose + caption_level: 3 + tablename: trips + nominal_col: trip_mode + row_grouping: primary_purpose + axis_label: Trip Mode Share + +- name: activitysim.workflows.steps.contrast.nominal_choice + in: + caption: Trip Mode Choice by Departure Time + caption_level: 3 + tablename: trips + nominal_col: trip_mode + row_grouping: depart + axis_label: Trip Mode Share + + +#### Trip Distance #### + +- name: activitysim.workflows.steps.reporting.section_title + in: + title: Trip Distance + +- name: activitysim.workflows.steps.contrast.attach_skim_data + in: + tablename: trips + otaz_col: origin + dtaz_col: destination + time_col: depart + skim_vars: '{distance_skim}' + +- name: activitysim.workflows.steps.contrast.transform_data + in: + tablename: trips + column: '{distance_skim}' + out: distance_to10_binned + censor: + left: 0 + right: 10 + cut: + bins: 20 + labels: midpoint + +- name: activitysim.workflows.steps.contrast.ordinal_distribution + in: + caption: Trip Distance by the Primary Purpose, <10 miles + caption_level: 3 + tablename: trips + ordinal_col: distance_to10_binned + facet_grouping: primary_purpose + plot_type: count + interpolate: step + value_format: "0.2f" + axis_label: Distance (to 10 miles) + +- name: activitysim.workflows.steps.contrast.trip_distance + in: + caption: Trip Distance by Primary Purpose, <10 miles + caption_level: 3 + grouping: primary_purpose + dist_bins: 20 + dist_skim_name: '{distance_skim}' + max_dist: 10 + +- name: activitysim.workflows.steps.contrast.trip_distance + in: + caption: Trip Distance by Primary Purpose + caption_level: 3 + grouping: primary_purpose + dist_bins: 20 + dist_skim_name: '{distance_skim}' + + +################################################################################ +report-save: +- name: activitysim.workflows.steps.reporting.save_report + in: + html_filename: "{workspace}/{example_name}/output-{tag}/report-{tag}.html" diff --git a/activitysim/workflows/sharrow-contrast/arc_full.yaml b/activitysim/workflows/sharrow-contrast/arc_full.yaml new file mode 100644 index 0000000000..9557c56f3a --- /dev/null +++ b/activitysim/workflows/sharrow-contrast/arc_full.yaml @@ -0,0 +1,33 @@ +context_parser: pypyr.parser.keyvaluepairs +steps: + +- description: Setting default workflow parameters + name: pypyr.steps.default + in: + defaults: + example_name: prototype_arc_full + workspace: workspace + create: True + compile: True + sharrow: True + legacy: True + reference: True + reference_asim_version: "1.0.4" + tag: '2022-06' + resume_after: _ + fast: False + compile_n_households: 1000 + main_n_households: 100000 + config_dirs: configs + data_dir: data + instrument: False +# workplace_zone_agg: +# caption: Workplaces by County +# district_id: county_id + distance_skim: SOV_FREE_DISTANCE + household_income: income_in_thousands + +- name: activitysim.workflows.steps.pype + in: + pype: + name: sharrow-contrast/_contrast_runner diff --git a/activitysim/workflows/sharrow-contrast/arc_mini.yaml b/activitysim/workflows/sharrow-contrast/arc_mini.yaml new file mode 100644 index 0000000000..7a1663c9e8 --- /dev/null +++ b/activitysim/workflows/sharrow-contrast/arc_mini.yaml @@ -0,0 +1,33 @@ +context_parser: pypyr.parser.keyvaluepairs +steps: + +- description: Setting default workflow parameters + name: pypyr.steps.default + in: + defaults: + example_name: prototype_arc + workspace: workspace + create: True + compile: False + sharrow: False + legacy: False + reference: False + reference_asim_version: "1.0.4" + tag: '2022-06' + resume_after: _ + fast: False + compile_n_households: 1000 + main_n_households: 1000 + config_dirs: configs + data_dir: data + instrument: False +# workplace_zone_agg: +# caption: Workplaces by County +# district_id: county_id + distance_skim: SOV_FREE_DISTANCE + household_income: income_in_thousands + +- name: activitysim.workflows.steps.pype + in: + pype: + name: sharrow-contrast/_contrast_runner diff --git a/activitysim/workflows/sharrow-contrast/arc_mp.yaml b/activitysim/workflows/sharrow-contrast/arc_mp.yaml new file mode 100644 index 0000000000..f494f6296e --- /dev/null +++ b/activitysim/workflows/sharrow-contrast/arc_mp.yaml @@ -0,0 +1,42 @@ +# +# arc_mp +# - activitysim workflow sharrow-contrast/arc_mp +# +context_parser: pypyr.parser.keyvaluepairs +steps: + +- description: Setting default workflow parameters + name: pypyr.steps.default + in: + defaults: + example_name: prototype_arc_full + workflow_name: sharrow-contrast/arc_mp + workspace: workspace + multiprocess: -4 + settings_file: settings_mp.yaml + chunk_size_pct_of_available: 0.75 + chunk_size_minimum_gb: 40 + create: True + compile: True + sharrow: True + legacy: True + reference: False + tag: + resume_after: + fast: False + compile_n_households: 5000 + main_n_households: 0 + instrument: False + memory_profile: True + trace_hh_id: + trace_od: + workplace_zone_agg: + caption: Workplaces by County + district_id: county_id + distance_skim: DIST + household_income: income + +- name: activitysim.workflows.steps.pype + in: + pype: + name: sharrow-contrast/_contrast_runner_chunked diff --git a/activitysim/workflows/sharrow-contrast/comprehensive.yaml b/activitysim/workflows/sharrow-contrast/comprehensive.yaml new file mode 100644 index 0000000000..737e2dc302 --- /dev/null +++ b/activitysim/workflows/sharrow-contrast/comprehensive.yaml @@ -0,0 +1,123 @@ +context_parser: pypyr.parser.keyvaluepairs +steps: + +- description: Setting default workflow parameters + name: pypyr.steps.default + in: + defaults: + workspace: workspace + create: True + compile: True + sharrow: True + legacy: True + reference: True + reference_asim_version: "1.0.4" + tag: + compile_n_households: 100 + main_n_households_singleprocess: 100 + main_n_households_multiprocess: 1000 + multiprocess: 8 + resume_after: + +- description: Ensure integer types for numbers + name: activitysim.workflows.steps.py + in: + label: Ensure integer types for numbers + py: | + save( + compile_n_households=int(compile_n_households), + main_n_households_singleprocess=int(main_n_households_singleprocess), + main_n_households_multiprocess=int(main_n_households_multiprocess), + multiprocess=int(multiprocess), + ) + +- description: Generate a tag based on date if tag is not given + name: activitysim.workflows.steps.py + in: + label: Generate tag + py: | + import time + if tag is None: + tag = time.strftime("%Y-%m-%d-Comprehensive") + save(tag=tag) + +- name: activitysim.workflows.steps.pype + description: Prototype MTC Model, Single Process, Zarr Disabled + skip: false + in: + pype: + name: sharrow-contrast/mtc_full + args: + workspace: '{workspace}' + create: '{create}' + compile: '{compile}' + sharrow: '{sharrow}' + legacy: '{legacy}' + reference: '{reference}' + disable_zarr: True + memory_profile: True + reference_asim_version: "{reference_asim_version}" + tag: '{tag}-SingleProcess' + resume_after: '{resume_after}' + compile_n_households: '{compile_n_households}' + main_n_households: '{main_n_households_singleprocess}' + +- name: activitysim.workflows.steps.pype + description: Prototype MTC Model, Single Process + skip: false + in: + pype: + name: sharrow-contrast/mtc_full + args: + workspace: '{workspace}' + create: False + compile: '{compile}' + sharrow: '{sharrow}' + legacy: '{legacy}' + reference: False + memory_profile: True + reference_asim_version: "{reference_asim_version}" + tag: '{tag}-SingleProcess-Zarr' + resume_after: '{resume_after}' + compile_n_households: '{compile_n_households}' + main_n_households: '{main_n_households_singleprocess}' + +- name: activitysim.workflows.steps.pype + description: Prototype MTC Model, Multiple Processes + skip: false + in: + pype: + name: sharrow-contrast/mtc_mp + args: + workspace: '{workspace}' + create: '{create}' + compile: '{compile}' + sharrow: '{sharrow}' + legacy: '{legacy}' + reference: '{reference}' + reference_asim_version: "{reference_asim_version}" + tag: '{tag}-MultiProcess' + resume_after: '{resume_after}' + multiprocess: '{multiprocess}' + compile_n_households: '{compile_n_households}' + main_n_households: '{main_n_households_multiprocess}' + +- name: activitysim.workflows.steps.pype + description: Prototype ARC Model, Single Process + skip: false + in: + pype: + name: sharrow-contrast/arc_full + args: + workspace: '{workspace}' + create: False + compile: '{compile}' + sharrow: '{sharrow}' + legacy: '{legacy}' + reference: '{reference}' + memory_profile: True + reference_asim_version: "{reference_asim_version}" + tag: '{tag}-SingleProcess' + resume_after: '{resume_after}' + compile_n_households: '{compile_n_households}' + main_n_households: '{main_n_households_singleprocess}' diff --git a/activitysim/workflows/sharrow-contrast/mtc_chunked.yaml b/activitysim/workflows/sharrow-contrast/mtc_chunked.yaml new file mode 100644 index 0000000000..943e486a30 --- /dev/null +++ b/activitysim/workflows/sharrow-contrast/mtc_chunked.yaml @@ -0,0 +1,50 @@ +# +# mtc_chunked +# +# This workflow runs the Prototype MTC model using full skims (1454 zones), +# in a chunked multi-process runner. Chunking is complicated and recommended +# only when running without chunking is infeasible due to excessive memory +# usage. +# + +context_parser: pypyr.parser.keyvaluepairs +steps: + +- description: Setting default workflow parameters + name: pypyr.steps.default + in: + defaults: + example_name: prototype_mtc_full + workflow_name: sharrow-contrast/mtc_chunked + multiprocess: 4 + chunk_size: 30_000_000_000 # 30 GB + chunk_training: True + create: False + compile: False + sharrow: True + legacy: True + reference: True + workspace: workspace + tag: '2022-06-complete' + resume_after: + fast: False + compile_n_households: 10000 + main_n_households: 0 # complete data + config_dirs: + - configs_mp + - configs + data_dir: data + instrument: False + memory_profile: True + trace_hh_id: + trace_od: + distance_skim: DIST + household_income: income + workplace_zone_agg: + caption: Workplaces by County + district_id: county_id + +- name: activitysim.workflows.steps.pype + in: + pype: + name: sharrow-contrast/_contrast_runner_chunked diff --git a/activitysim/workflows/sharrow-contrast/mtc_comprehensive.yaml b/activitysim/workflows/sharrow-contrast/mtc_comprehensive.yaml new file mode 100644 index 0000000000..685bb5a577 --- /dev/null +++ b/activitysim/workflows/sharrow-contrast/mtc_comprehensive.yaml @@ -0,0 +1,113 @@ +# +# mtc_comprehensive +# +# This workflow runs the Prototype MTC model using full skims (1454 zones), +# running it multiple times in different configurations (single process, +# single process with zarr compression, multi-process). +# + +context_parser: pypyr.parser.keyvaluepairs +steps: + +- description: Setting default workflow parameters + name: pypyr.steps.default + in: + defaults: + workspace: workspace + create: True + compile: True + sharrow: True + legacy: True + reference: True + reference_asim_version: "1.0.4" + tag: + compile_n_households: 2000 + main_n_households_singleprocess: 100000 + main_n_households_multiprocess: 0 + multiprocess: -4 + resume_after: + +- description: Ensure integer types for numbers + name: activitysim.workflows.steps.py + in: + label: Ensure integer types for numbers + py: | + save( + compile_n_households=int(compile_n_households), + main_n_households_singleprocess=int(main_n_households_singleprocess), + main_n_households_multiprocess=int(main_n_households_multiprocess), + multiprocess=int(multiprocess), + ) + +- description: Generate a tag based on date if tag is not given + name: activitysim.workflows.steps.py + in: + label: Generate tag + py: | + import time + if tag is None: + tag = time.strftime("%Y-%m-%d-Comprehensive") + save(tag=tag) + +- name: activitysim.workflows.steps.pype + description: Prototype MTC Model, Single Process, Zarr Disabled + skip: false + in: + pype: + name: sharrow-contrast/mtc_full + args: + workflow_name: sharrow-contrast/mtc_full (zarr disabled) + workspace: '{workspace}' + create: '{create}' + compile: '{compile}' + sharrow: '{sharrow}' + legacy: '{legacy}' + reference: '{reference}' + disable_zarr: True + memory_profile: True + reference_asim_version: "{reference_asim_version}" + tag: '{tag}-SingleProcess' + resume_after: '{resume_after}' + compile_n_households: '{compile_n_households}' + main_n_households: '{main_n_households_singleprocess}' + +- name: activitysim.workflows.steps.pype + description: Prototype MTC Model, Single Process + skip: false + in: + pype: + name: sharrow-contrast/mtc_full + args: + workflow_name: sharrow-contrast/mtc_full (zarr enabled) + workspace: '{workspace}' + create: False + compile: '{compile}' + sharrow: '{sharrow}' + legacy: '{legacy}' + reference: False + memory_profile: True + reference_asim_version: "{reference_asim_version}" + tag: '{tag}-SingleProcess-Zarr' + resume_after: '{resume_after}' + compile_n_households: '{compile_n_households}' + main_n_households: '{main_n_households_singleprocess}' + +- name: activitysim.workflows.steps.pype + description: Prototype MTC Model, Multiple Processes + skip: false + in: + pype: + name: sharrow-contrast/mtc_mp + args: + workspace: '{workspace}' + create: '{create}' + compile: '{compile}' + sharrow: '{sharrow}' + legacy: '{legacy}' + reference: '{reference}' + reference_asim_version: "{reference_asim_version}" + tag: '{tag}-MultiProcess' + resume_after: '{resume_after}' + multiprocess: '{multiprocess}' + compile_n_households: '{compile_n_households}' + main_n_households: '{main_n_households_multiprocess}' diff --git a/activitysim/workflows/sharrow-contrast/mtc_full.yaml b/activitysim/workflows/sharrow-contrast/mtc_full.yaml new file mode 100644 index 0000000000..404692eedc --- /dev/null +++ b/activitysim/workflows/sharrow-contrast/mtc_full.yaml @@ -0,0 +1,44 @@ +# +# mtc_full +# +# This workflow runs the Prototype MTC model using full skims (1454 zones), +# in a single process runner. It is well suited for conducting timing and +# memory usage studies, as well as smaller model quality and validation +# studies. For running the full population, consider running in multiprocess +# mode with the `mtc_mp` workflow. +# + +context_parser: pypyr.parser.keyvaluepairs +steps: + +- description: Setting default workflow parameters + name: pypyr.steps.default + in: + defaults: + example_name: prototype_mtc_full + workflow_name: sharrow-contrast/mtc_full + workspace: workspace + create: True + compile: True + sharrow: True + legacy: True + reference: True + reference_asim_version: "1.0.4" + tag: + resume_after: _ + fast: False + compile_n_households: 1000 + main_n_households: 100000 + config_dirs: configs + data_dir: data + instrument: False + workplace_zone_agg: + caption: Workplaces by County + district_id: county_id + distance_skim: DIST + household_income: income + +- name: activitysim.workflows.steps.pype + in: + pype: + name: sharrow-contrast/_contrast_runner diff --git a/activitysim/workflows/sharrow-contrast/mtc_mini.yaml b/activitysim/workflows/sharrow-contrast/mtc_mini.yaml new file mode 100644 index 0000000000..12bd36a8b5 --- /dev/null +++ b/activitysim/workflows/sharrow-contrast/mtc_mini.yaml @@ -0,0 +1,44 @@ +# +# mtc_mini +# +# This workflow runs the Prototype MTC model using mini skims (25 zones), +# in a single process runner. It is for mainly for rapid testing of the code +# and specification files for errors, not for policy analysis. +# + +context_parser: pypyr.parser.keyvaluepairs +steps: + +- description: Setting default workflow parameters + name: pypyr.steps.default + in: + defaults: + example_name: prototype_mtc + workflow_name: sharrow-contrast/mtc_mini + workspace: workspace # this is the directory where model runs are stored + create: True + compile: True + sharrow: True + legacy: True + reference: True + tag: + resume_after: + fast: False + compile_n_households: 5000 + main_n_households: 100000 + config_dirs: configs + data_dir: data + instrument: False + memory_profile: True + trace_hh_id: + trace_od: + workplace_zone_agg: + caption: Workplaces by County + district_id: county_id + distance_skim: DIST + household_income: income + +- name: activitysim.workflows.steps.pype + in: + pype: + name: sharrow-contrast/_contrast_runner diff --git a/activitysim/workflows/sharrow-contrast/mtc_mp.yaml b/activitysim/workflows/sharrow-contrast/mtc_mp.yaml new file mode 100644 index 0000000000..f5916bc71c --- /dev/null +++ b/activitysim/workflows/sharrow-contrast/mtc_mp.yaml @@ -0,0 +1,59 @@ +# +# mtc_mp +# +# This workflow runs the Prototype MTC model using full skims (1454 zones), +# in a multi-process runner. It is well suited for large runs with the full +# population. To do detailed analysis of performance or debugging, consider +# running in single-process mode with the `mtc_full` workflow. +# + +context_parser: pypyr.parser.keyvaluepairs +steps: + +- description: Setting default workflow parameters + name: pypyr.steps.default + in: + defaults: + example_name: prototype_mtc_full + workflow_name: sharrow-contrast/mtc_mp + workspace: workspace + multiprocess: 8 + chunk_size: 0 + create: True + compile: True + sharrow: True + legacy: True + reference: True + reference_asim_version: "1.0.4" + tag: + resume_after: + fast: False + compile_n_households: 2000 + main_n_households: 100000 + config_dirs: + - configs_mp + - configs + data_dir: data + instrument: False + workplace_zone_agg: + caption: Workplaces by County + district_id: county_id + distance_skim: DIST + household_income: income + +- description: Ensure integer types for numbers + name: activitysim.workflows.steps.py + in: + label: Ensure integer types for numbers + py: | + save( + compile_n_households=int(compile_n_households), + main_n_households=int(main_n_households), + multiprocess=int(multiprocess), + chunk_size=int(chunk_size), + ) + +- name: activitysim.workflows.steps.pype + in: + pype: + name: sharrow-contrast/_contrast_runner diff --git a/activitysim/workflows/sharrow-contrast/psrc_mini.yaml b/activitysim/workflows/sharrow-contrast/psrc_mini.yaml new file mode 100644 index 0000000000..06e540d2ef --- /dev/null +++ b/activitysim/workflows/sharrow-contrast/psrc_mini.yaml @@ -0,0 +1,28 @@ +context_parser: pypyr.parser.keyvaluepairs +steps: + +- description: Setting default workflow parameters + name: pypyr.steps.default + in: + defaults: + example_name: placeholder_psrc + workspace: workspace + create: True + compile: True + sharrow: True + legacy: True + tag: '2022-04' + resume_after: _ + fast: False + compile_n_households: 100 + main_n_households: 100000 + config_dirs: + #- configs_skip_accessibility + - configs + data_dir: data + instrument: False + +- name: activitysim.workflows.steps.pype + in: + pype: + name: sharrow-contrast/_contrast_runner diff --git a/activitysim/workflows/sharrow-contrast/sandag_1zone_complete.yaml b/activitysim/workflows/sharrow-contrast/sandag_1zone_complete.yaml new file mode 100644 index 0000000000..925253e5bb --- /dev/null +++ b/activitysim/workflows/sharrow-contrast/sandag_1zone_complete.yaml @@ -0,0 +1,38 @@ +context_parser: pypyr.parser.keyvaluepairs +steps: + +- description: Setting default workflow parameters + name: pypyr.steps.default + in: + defaults: + example_name: placeholder_sandag_1_zone_full + multiprocess: 4 + chunk_size_minimum_gb: 30 + chunk_training: True + create: False + compile: True + sharrow: True + legacy: False + reference: False + workspace: workspace + tag: '2022-07' + resume_after: + fast: False + compile_n_households: 2000 + main_n_households: 0 # complete data + config_dirs: + - configs_1_zone + - prototype_mtc/configs + data_dir: data_1 + settings_file: settings_mp.yaml + instrument: False + memory_profile: True + trace_hh_id: + trace_od: + distance_skim: DIST + household_income: income + +- name: activitysim.workflows.steps.pype + in: + pype: + name: sharrow-contrast/_contrast_runner_chunked diff --git a/activitysim/workflows/sharrow-contrast/sandag_1zone_full.yaml b/activitysim/workflows/sharrow-contrast/sandag_1zone_full.yaml new file mode 100644 index 0000000000..3aa5d2f08d --- /dev/null +++ b/activitysim/workflows/sharrow-contrast/sandag_1zone_full.yaml @@ -0,0 +1,39 @@ +context_parser: pypyr.parser.keyvaluepairs +steps: + +- description: Setting default workflow parameters + name: pypyr.steps.default + in: + defaults: + example_name: placeholder_sandag_1_zone_full + workspace: workspace + create: True + compile: True + sharrow: True + legacy: True + reference: False + tag: '2022-07-1zone-1process' + resume_after: + fast: False + compile_n_households: 10000 + main_n_households: 100000 + config_dirs: + - configs_1_zone + - prototype_mtc/configs + data_dir: data_1 + instrument: False + memory_profile: True + trace_hh_id: + trace_od: + workplace_zone_agg: + caption: Workplaces by County + district_id: county_id + distance_skim: DIST + household_income: income + chunk_training: True + chunk_training_mode: training + +- name: activitysim.workflows.steps.pype + in: + pype: + name: sharrow-contrast/_contrast_runner diff --git a/activitysim/workflows/sharrow-contrast/sandag_1zone_mini.yaml b/activitysim/workflows/sharrow-contrast/sandag_1zone_mini.yaml new file mode 100644 index 0000000000..3615867c46 --- /dev/null +++ b/activitysim/workflows/sharrow-contrast/sandag_1zone_mini.yaml @@ -0,0 +1,37 @@ +context_parser: pypyr.parser.keyvaluepairs +steps: + +- description: Setting default workflow parameters + name: pypyr.steps.default + in: + defaults: + example_name: placeholder_sandag_1_zone + workspace: workspace + create: True + compile: True + sharrow: True + legacy: True + reference: False + tag: '2022-06-1zone' + resume_after: + fast: False + compile_n_households: 10000 + main_n_households: 100000 + config_dirs: + - configs_1_zone + - prototype_mtc/configs + data_dir: data_1 + instrument: False + memory_profile: True + trace_hh_id: + trace_od: + workplace_zone_agg: + caption: Workplaces by County + district_id: county_id + distance_skim: DIST + household_income: income + +- name: activitysim.workflows.steps.pype + in: + pype: + name: sharrow-contrast/_contrast_runner diff --git a/activitysim/workflows/sharrow-contrast/sandag_1zone_mp.yaml b/activitysim/workflows/sharrow-contrast/sandag_1zone_mp.yaml new file mode 100644 index 0000000000..f08056ed03 --- /dev/null +++ b/activitysim/workflows/sharrow-contrast/sandag_1zone_mp.yaml @@ -0,0 +1,46 @@ +# +# sandag_1zone_mp +# - activitysim workflow sharrow-contrast/sandag_1zone_mp +# +context_parser: pypyr.parser.keyvaluepairs +steps: + +- description: Setting default workflow parameters + name: pypyr.steps.default + in: + defaults: + example_name: placeholder_sandag_1_zone_full + workflow_name: sharrow-contrast/sandag_1zone_mp + workspace: workspace + multiprocess: -4 + settings_file: settings_mp.yaml + chunk_size_pct_of_available: 0.75 + chunk_size_minimum_gb: 40 + create: True + compile: True + sharrow: True + legacy: True + reference: False + tag: + resume_after: + fast: False + compile_n_households: 1000 + main_n_households: 0 + config_dirs: + - configs_1_zone + - prototype_mtc/configs + data_dir: data_1 + instrument: False + memory_profile: True + trace_hh_id: + trace_od: + workplace_zone_agg: + caption: Workplaces by County + district_id: county_id + distance_skim: DIST + household_income: income + +- name: activitysim.workflows.steps.pype + in: + pype: + name: sharrow-contrast/_contrast_runner_chunked diff --git a/activitysim/workflows/sharrow-contrast/sandag_2zone_mini.yaml b/activitysim/workflows/sharrow-contrast/sandag_2zone_mini.yaml new file mode 100644 index 0000000000..03b59e5bd8 --- /dev/null +++ b/activitysim/workflows/sharrow-contrast/sandag_2zone_mini.yaml @@ -0,0 +1,46 @@ +# +# sandag_2zone_mini +# +# This workflow runs the Placeholder SANDAG model using the 2-Zone system +# in a single process runner. It is for mainly for rapid testing of the code +# and specification files for errors, not for policy analysis. +# +# > activitysim workflow sharrow-contrast/sandag_2zone_mini + +context_parser: pypyr.parser.keyvaluepairs +steps: + +- description: Setting default workflow parameters + name: pypyr.steps.default + in: + defaults: + example_name: placeholder_sandag_2_zone + workspace: workspace + create: True + compile: True + sharrow: True + legacy: True + reference: False + tag: '2022-07-2zone' + resume_after: + fast: False + compile_n_households: 100 + main_n_households: 100000 + config_dirs: + - configs_2_zone + - placeholder_psrc/configs + data_dir: data_2 + instrument: False + memory_profile: True + trace_hh_id: + trace_od: + workplace_zone_agg: + caption: Workplaces by County + district_id: county_id + distance_skim: DIST + household_income: income + +- name: activitysim.workflows.steps.pype + in: + pype: + name: sharrow-contrast/_contrast_runner diff --git a/activitysim/workflows/sharrow-contrast/sandag_3zone_mini.yaml b/activitysim/workflows/sharrow-contrast/sandag_3zone_mini.yaml new file mode 100644 index 0000000000..9644758e52 --- /dev/null +++ b/activitysim/workflows/sharrow-contrast/sandag_3zone_mini.yaml @@ -0,0 +1,46 @@ +# +# sandag_3zone_mini +# +# This workflow runs the Placeholder SANDAG model using the 3-Zone system +# in a single process runner. It is for mainly for rapid testing of the code +# and specification files for errors, not for policy analysis. +# +# > activitysim workflow sharrow-contrast/sandag_3zone_mini + +context_parser: pypyr.parser.keyvaluepairs +steps: + +- description: Setting default workflow parameters + name: pypyr.steps.default + in: + defaults: + example_name: placeholder_sandag_3_zone + workspace: workspace + create: True + compile: True + sharrow: True + legacy: True + reference: False + tag: + resume_after: + fast: False + compile_n_households: 100 + main_n_households: 0 + config_dirs: + - configs_3_zone + - prototype_mtc/configs + data_dir: data_3 + instrument: False + memory_profile: False + trace_hh_id: + trace_od: + workplace_zone_agg: + caption: Workplaces by County + district_id: county_id + distance_skim: DIST + household_income: income + +- name: activitysim.workflows.steps.pype + in: + pype: + name: sharrow-contrast/_contrast_runner diff --git a/activitysim/workflows/sharrow-contrast/sandag_xborder_mini.yaml b/activitysim/workflows/sharrow-contrast/sandag_xborder_mini.yaml new file mode 100644 index 0000000000..6c44b6455d --- /dev/null +++ b/activitysim/workflows/sharrow-contrast/sandag_xborder_mini.yaml @@ -0,0 +1,49 @@ +# +# sandag_xborder_mini +# +# This workflow runs the Prototype SANDAG cross-border model in a single- +# process runner. +# +# > activitysim workflow sharrow-contrast/sandag_xborder_mini + +context_parser: pypyr.parser.keyvaluepairs +steps: + +- description: Setting default workflow parameters + name: pypyr.steps.default + in: + defaults: + example_name: prototype_sandag_xborder + workspace: workspace + create: True + compile: True + sharrow: True + legacy: True + reference: False + tag: + resume_after: + fast: False + compile_n_households: 100 + main_n_households: 0 + config_dirs: + - configs + data_dir: data + ext_dirs: + - extensions + instrument: False + memory_profile: False + trace_hh_id: + trace_od: + distance_skim: SOV_NT_M_DIST + household_income: + report_skip: + - auto ownership + - cdap + - tour mode by composition + - work location + - school location + +- name: activitysim.workflows.steps.pype + in: + pype: + name: sharrow-contrast/_contrast_runner diff --git a/activitysim/workflows/steps/__init__.py b/activitysim/workflows/steps/__init__.py new file mode 100644 index 0000000000..c4ac3b54bf --- /dev/null +++ b/activitysim/workflows/steps/__init__.py @@ -0,0 +1,31 @@ +# from . import archive_outputs +from pypyr.context import Context +from pypyr.errors import KeyNotInContextError + +from . import cmd, py + + +def get_formatted_or_default(self: Context, key: str, default): + try: + return self.get_formatted(key) + except (KeyNotInContextError, KeyError): + return default + except TypeError: + return self.get(key) + except Exception as err: + raise ValueError(f"extracting {key} from context") from err + + +Context.get_formatted_or_default = get_formatted_or_default + + +def get_formatted_or_raw(self: Context, key: str): + try: + return self.get_formatted(key) + except TypeError: + return self.get(key) + except Exception as err: + raise ValueError(f"extracting {key} from context") from err + + +Context.get_formatted_or_raw = get_formatted_or_raw diff --git a/activitysim/workflows/steps/chunk_sizing.py b/activitysim/workflows/steps/chunk_sizing.py new file mode 100644 index 0000000000..ee0ceca699 --- /dev/null +++ b/activitysim/workflows/steps/chunk_sizing.py @@ -0,0 +1,50 @@ +import warnings + +import psutil + +from .progression import reset_progress_step +from .wrapping import workstep + + +@workstep(updates_context=True) +def chunk_sizing( + chunk_size=None, + chunk_size_pct_of_available=None, + chunk_size_pct_of_total=None, + chunk_size_minimum_gb=0, +): + reset_progress_step(description="Figuring chunk size") + + # by default, if neither pct_of_available or pct_of_total is given, + # and chunk_size is not set explicitly, then use 85% of available RAM + if chunk_size_pct_of_available is None and chunk_size_pct_of_total is None: + chunk_size_pct_of_available = 0.85 + + vm = psutil.virtual_memory() + available_ram = vm.available + total_ram = vm.total + + # if chunk size is set explicitly, use it without regard to other settings + if chunk_size is None: + + if chunk_size_pct_of_available is not None: + if chunk_size_pct_of_available > 1: + chunk_size_pct_of_available /= 100 + chunk_size = int(available_ram * chunk_size_pct_of_available) + elif chunk_size_pct_of_total is not None: + if chunk_size_pct_of_total > 1: + chunk_size_pct_of_total /= 100 + chunk_size = int(total_ram * chunk_size_pct_of_total) + + min_chunk_size = int(chunk_size_minimum_gb * 1e9) + if chunk_size < min_chunk_size: + chunk_size = min_chunk_size + + if chunk_size > total_ram: + warnings.warn( + f"chunk size of {chunk_size/ 2**30:.2f}GB exceeds " + f"total RAM of {total_ram/ 2**30:.2f}" + ) + + out = dict(chunk_size=chunk_size) + return out diff --git a/activitysim/workflows/steps/cmd/__init__.py b/activitysim/workflows/steps/cmd/__init__.py new file mode 100644 index 0000000000..e62b19e851 --- /dev/null +++ b/activitysim/workflows/steps/cmd/__init__.py @@ -0,0 +1,54 @@ +"""pypyr step that executes a cmd as a sub-process. + +You cannot use things like exit, return, shell pipes, filename wildcards, +environment,variable expansion, and expansion of ~ to a user’s home +directory. +""" +import logging + +from .dsl import CmdStep + +# logger means the log level will be set correctly +logger = logging.getLogger(__name__) + + +def run_step(context): + """Run command, program or executable. + + Context is a dictionary or dictionary-like. + + Context must contain the following keys: + cmd: <> (command + args to execute.) + + OR, as a dict + cmd: + run: str. mandatory. <> command + args to execute. + save: bool. defaults False. save output to cmdOut. + + Will execute the command string in the shell as a sub-process. + Escape curly braces: if you want a literal curly brace, double it like + {{ or }}. + + If save is True, will save the output to context as follows: + cmdOut: + returncode: 0 + stdout: 'stdout str here. None if empty.' + stderr: 'stderr str here. None if empty.' + + cmdOut.returncode is the exit status of the called process. Typically 0 + means OK. A negative value -N indicates that the child was terminated by + signal N (POSIX only). + + context['cmd'] will interpolate anything in curly braces for values + found in context. So if your context looks like this: + key1: value1 + key2: value2 + cmd: mything --arg1 {key1} + + The cmd passed to the shell will be "mything --arg value1" + """ + logger.debug("started") + + CmdStep(name=__name__, context=context).run_step(is_shell=False) + + logger.debug("done") diff --git a/activitysim/workflows/steps/cmd/dsl.py b/activitysim/workflows/steps/cmd/dsl.py new file mode 100644 index 0000000000..1d05a7d3b6 --- /dev/null +++ b/activitysim/workflows/steps/cmd/dsl.py @@ -0,0 +1,167 @@ +"""pypyr step yaml definition for commands - domain specific language.""" +import logging +import os +import shlex +import subprocess +import sys +import time + +import pypyr.errors +from pypyr.config import config +from pypyr.errors import ContextError +from pypyr.utils import types + +from ..progression import reset_progress_step + +# logger means the log level will be set correctly +logger = logging.getLogger(__name__) + +RED = "\033[91m" +END = "\033[0m" + + +def stream_process(process, label): + go = True + while go: + go = process.poll() is None + for line in process.stdout: + print(line.decode().rstrip()) + for line in process.stderr: + print(RED + line.decode().rstrip() + END, file=sys.stderr) + + +class CmdStep: + """A pypyr step that represents a command runner step. + + This models a step that takes config like this: + cmd: <> + + OR, as a dict + cmd: + run: str. mandatory. command + args to execute. + save: bool. defaults False. save output to cmdOut. + + If save is True, will save the output to context as follows: + cmdOut: + returncode: 0 + stdout: 'stdout str here. None if empty.' + stderr: 'stderr str here. None if empty.' + + cmdOut.returncode is the exit status of the called process. Typically 0 + means OK. A negative value -N indicates that the child was terminated by + signal N (POSIX only). + + The run_step method does the actual work. init loads the yaml. + """ + + def __init__(self, name, context): + """Initialize the CmdStep. + + The step config in the context dict looks like this: + cmd: <> + + OR, as a dict + cmd: + run: str. mandatory. command + args to execute. + save: bool. optional. defaults False. save output to cmdOut. + cwd: str/path. optional. if specified, change the working + directory just for the duration of the command. + + Args: + name: Unique name for step. Likely __name__ of calling step. + context: pypyr.context.Context. Look for config in this context + instance. + + """ + assert name, "name parameter must exist for CmdStep." + assert context, "context param must exist for CmdStep." + # this way, logs output as the calling step, which makes more sense + # to end-user than a mystery steps.dsl.blah logging output. + self.logger = logging.getLogger(name) + + context.assert_key_has_value(key="cmd", caller=name) + + self.context = context + + cmd_config = context.get_formatted("cmd") + + if isinstance(cmd_config, str): + self.cmd_text = cmd_config + self.cwd = None + self.logger.debug("Processing command string: %s", cmd_config) + self.label = cmd_config + elif isinstance(cmd_config, dict): + context.assert_child_key_has_value(parent="cmd", child="run", caller=name) + + self.cmd_text = cmd_config["run"] + self.label = cmd_config.get("label", self.cmd_text) + self.conda_path = cmd_config.get("conda_path", None) + + cwd_string = cmd_config.get("cwd", None) + if cwd_string: + self.cwd = cwd_string + self.logger.debug( + "Processing command string in dir " "%s: %s", + self.cwd, + self.cmd_text, + ) + else: + self.cwd = None + self.logger.debug("Processing command string: %s", self.cmd_text) + + else: + raise ContextError( + f"{name} cmd config should be either a simple " + "string cmd='mycommandhere' or a dictionary " + "cmd={'run': 'mycommandhere', 'save': False}." + ) + + def run_step(self, is_shell): + """Run a command. + + Runs a program or executable. If is_shell is True, executes the command + through the shell. + + Args: + is_shell: bool. defaults False. Set to true to execute cmd through + the default shell. + """ + assert is_shell is not None, "is_shell param must exist for CmdStep." + conda_path = self.conda_path or sys.exec_prefix + + # why? If shell is True, it is recommended to pass args as a string + # rather than as a sequence. + # But not on windows, because windows wants strings, not sequences. + if is_shell or config.is_windows: + args = f'conda run -p "{conda_path}" ' + self.cmd_text + else: + args = shlex.split(self.cmd_text) + args = ["conda", "run", "-p", conda_path] + list(args) # TODO windows? + + reset_progress_step(description=f"{self.label}", prefix="[bold green]") + + env = os.environ.copy() + pythonpath = env.pop("PYTHONPATH", None) + + process = subprocess.Popen( + args, + shell=is_shell, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + cwd=self.cwd, + env=env, + ) + stream_process(process, self.label) + self.context["cmdOut"] = { + "returncode": process.returncode, + } + + # don't swallow the error, because it's the Step swallow decorator + # responsibility to decide to ignore or not. + if process.returncode: + raise subprocess.CalledProcessError( + process.returncode, + process.args, + process.stdout, + process.stderr, + ) diff --git a/activitysim/workflows/steps/contrast/__init__.py b/activitysim/workflows/steps/contrast/__init__.py new file mode 100644 index 0000000000..5ef49ba904 --- /dev/null +++ b/activitysim/workflows/steps/contrast/__init__.py @@ -0,0 +1,2 @@ +from .load_skims import load_skims +from .load_tables import load_tables diff --git a/activitysim/workflows/steps/contrast/attach_skim_data.py b/activitysim/workflows/steps/contrast/attach_skim_data.py new file mode 100644 index 0000000000..b872afd249 --- /dev/null +++ b/activitysim/workflows/steps/contrast/attach_skim_data.py @@ -0,0 +1,89 @@ +import logging + +import numpy as np +import pandas as pd + +from ..progression import reset_progress_step +from ..wrapping import workstep + +logger = logging.getLogger(__name__) + + +def _as_int(x): + if x.dtype.kind == "i": + return x + else: + return x.astype(np.int32) + + +@workstep(updates_context=True) +def attach_skim_data( + tablesets, + skims, + skim_vars, + tablename, + otaz_col, + dtaz_col, + time_col=None, + rename_skim_vars=None, +) -> dict: + if isinstance(skim_vars, str): + skim_vars = [skim_vars] + if len(skim_vars) == 1: + skim_vars_note = skim_vars[0] + else: + skim_vars_note = f"{len(skim_vars)} skim vars" + + reset_progress_step( + description=f"attach skim data / {tablename} <- {skim_vars_note}" + ) + + if not isinstance(skims, dict): + skims = {i: skims for i in tablesets.keys()} + + for key, tableset in tablesets.items(): + # skim_subset = skims[key][list(skims[key].coords) + skim_vars] + skim_subset = skims[key][skim_vars] + + otag = "omaz" if "omaz" in skims[key].coords else "otaz" + dtag = "dmaz" if "dmaz" in skims[key].coords else "dtaz" + + zone_ids = tableset["land_use"].index + if ( + zone_ids.is_monotonic_increasing + and zone_ids[-1] == len(zone_ids) + zone_ids[0] - 1 + ): + offset = zone_ids[0] + looks = [ + _as_int(tableset[tablename][otaz_col].rename(otag) - offset), + _as_int(tableset[tablename][dtaz_col].rename(dtag) - offset), + ] + else: + remapper = dict(zip(zone_ids, pd.RangeIndex(len(zone_ids)))) + looks = [ + _as_int(tableset[tablename][otaz_col].rename(otag).apply(remapper.get)), + _as_int(tableset[tablename][dtaz_col].rename(dtag).apply(remapper.get)), + ] + if "time_period" in skim_subset.dims: + if time_col is None: + raise KeyError("time_period in skims to slice but time_col is missing") + looks.append( + tableset[tablename][time_col] + .apply(skims[key].attrs["time_period_imap"].get) + .rename("time_period"), + ) + look = pd.concat(looks, axis=1) + try: + out = skim_subset.iat.df(look) + except KeyError as err: + # KeyError is triggered when reading TAZ data from MAZ-enabled skims + lookr = {i: look[i].values for i in look.columns} + out = skims[key].iat(**lookr, _names=skim_vars).to_dataframe() + out = out.set_index(tablesets[key][tablename].index) + if rename_skim_vars is not None: + if isinstance(rename_skim_vars, str): + rename_skim_vars = [rename_skim_vars] + out = out.rename(columns=dict(zip(skim_vars, rename_skim_vars))) + tablesets[key][tablename] = tablesets[key][tablename].assign(**out) + + return dict(tablesets=tablesets) diff --git a/activitysim/workflows/steps/contrast/composite_log.py b/activitysim/workflows/steps/contrast/composite_log.py new file mode 100644 index 0000000000..81c90c599a --- /dev/null +++ b/activitysim/workflows/steps/contrast/composite_log.py @@ -0,0 +1,72 @@ +import os + +import pandas as pd + +from ..progression import reset_progress_step +from ..wrapping import workstep + + +def to_csv_safe(obj, filename, *args, **kwargs): + """ + Write a csv as normal, changing the filename if a permission error occurs. + """ + try: + obj.to_csv(filename, *args, **kwargs) + except PermissionError: + n = 1 + f1, f2 = os.path.splitext(filename) + while os.path.exists(f"{f1} ({n}){f2}"): + n += 1 + obj.to_csv(f"{f1} ({n}){f2}", *args, **kwargs) + + +@workstep(updates_context=True) +def composite_log( + tag, + archive_dir, + compares=("compile", "sharrow", "legacy", "reference"), +) -> dict: + + reset_progress_step(description="composite timing and memory logs") + + timings = {} + compares = list(compares) + for t in compares: + filename = f"{archive_dir}/output-{t}/log/timing_log.csv" + if os.path.exists(filename): + df = pd.read_csv(filename) + df = df.set_index("model_name")["seconds"] + timings[t] = df.loc[~df.index.duplicated()] + if timings: + composite_timing = pd.concat(timings, axis=1) + to_csv_safe(composite_timing, f"{archive_dir}/combined_timing_log-{tag}.csv") + mems = {} + for t in compares: + filename = f"{archive_dir}/output-{t}/log/mem.csv" + if os.path.exists(filename): + df = pd.read_csv(filename) + df = df.set_index("event")[["rss", "full_rss", "uss"]] + mems[t] = df.loc[~df.index.duplicated()] + if mems: + composite_mem = pd.concat(mems, axis=1) + to_csv_safe(composite_mem, f"{archive_dir}/combined_mem_log-{tag}.csv") + peaks = {} + for t in compares: + filename = f"{archive_dir}/output-{t}/log/memory_profile.csv" + if os.path.exists(filename): + df = pd.read_csv(filename) + df["time"] = pd.to_datetime(df["time"]) + peak_by_event = ( + df.groupby("event")[["rss", "full_rss", "uss", "time"]] + .max() + .sort_values("time") + ) + peaks[t] = peak_by_event[["rss", "full_rss", "uss"]] + if peaks: + composite_peaks = pd.concat(peaks, axis=1) + to_csv_safe(composite_peaks, f"{archive_dir}/combined_mem_peak-{tag}.csv") + return dict( + combined_timing_log=f"{archive_dir}/combined_timing_log-{tag}.csv", + combined_mem_log=f"{archive_dir}/combined_mem_log-{tag}.csv", + combined_peakmem_log=f"{archive_dir}/combined_mem_peak-{tag}.csv", + ) diff --git a/activitysim/workflows/steps/contrast/contrast_setup.py b/activitysim/workflows/steps/contrast/contrast_setup.py new file mode 100644 index 0000000000..b907730672 --- /dev/null +++ b/activitysim/workflows/steps/contrast/contrast_setup.py @@ -0,0 +1,56 @@ +import multiprocessing +import time + +from ..progression import reset_progress_step +from ..wrapping import workstep + + +@workstep(updates_context=True) +def contrast_setup( + example_name, + tag=None, + compile=True, + sharrow=True, + legacy=True, + resume_after=True, + fast=True, + reference=None, + reference_asim_version="0.0.0", + multiprocess=0, + chunk_training_mode=None, + main_n_households=None, +): + reset_progress_step(description="Constrast Setup") + if tag is None: + tag = time.strftime("%Y-%m-%d-%H%M%S") + contrast = sharrow and legacy + + flags = [] + if resume_after: + flags.append(f" -r {resume_after}") + if fast: + flags.append("--fast") + + out = dict(tag=tag, contrast=contrast, flags=" ".join(flags)) + if isinstance(reference, str) and "." in reference: + out["reference_asim_version"] = reference + out["reference"] = True + out["relabel_tablesets"] = {"reference": f"v{reference_asim_version}"} + multiprocess = int(multiprocess) + out["is_multiprocess"] = (multiprocess > 1) or (multiprocess < 0) + if multiprocess >= 0: + out["num_processes"] = multiprocess + else: + # when negative, count the number of cpu cores, and run on all + # cores except the absolute value of `multiprocess`, so e.g. + # if -2, then run on all cores except 2 (but at least 1). + out["num_processes"] = multiprocessing.cpu_count() + multiprocess + if out["num_processes"] < 1: + out["num_processes"] = 1 + + if chunk_training_mode and chunk_training_mode != "disabled": + out["chunk_application_mode"] = "production" + else: + out["chunk_application_mode"] = chunk_training_mode + + return out diff --git a/activitysim/workflows/steps/contrast/data_inventory.py b/activitysim/workflows/steps/contrast/data_inventory.py new file mode 100644 index 0000000000..a3ff172a31 --- /dev/null +++ b/activitysim/workflows/steps/contrast/data_inventory.py @@ -0,0 +1,76 @@ +import logging + +import altair as alt +import pandas as pd +import xmle +from pypyr.context import Context +from pypyr.errors import KeyNotInContextError + +from ..error_handler import error_logging +from ..progression import reset_progress_step + +logger = logging.getLogger(__name__) + + +@error_logging +def run_step(context: Context) -> None: + reset_progress_step(description="report model inventory") + + context.assert_key_has_value(key="report", caller=__name__) + report = context.get("report") + fig = context.get("fig") + tab = context.get("tab") + + tablesets = context.get("tablesets") + skims = context.get("skims") + try: + title = context.get_formatted("title") + except (KeyError, KeyNotInContextError): + title = "Data Inventory" + + lens = {} + dtypes = {} + for source, tableset in tablesets.items(): + lens[source] = {} + dtypes[source] = {} + for tablename, tabledata in tableset.items(): + lens[source][tablename] = len(tabledata) + for k, kt in tabledata.dtypes.items(): + dtypes[source][tablename, k] = kt + + with report: + report << f"## {title}" + + with report: + with pd.option_context( + "display.max_rows", 1_000_000, "display.max_columns", 10_000 + ): + report << tab("Table Row Count", level=3) + report << pd.DataFrame(lens).applymap( + lambda x: f"{x:,}" if isinstance(x, int) else x + ) + + with report: + with pd.option_context( + "display.max_rows", 1_000_000, "display.max_columns", 10_000 + ): + report << tab("Table Contents", level=3) + dtypes_table = ( + pd.DataFrame(dtypes).rename_axis(index=["table", "column"]).fillna("") + ) + dtypes_table[""] = pd.Series( + (dtypes_table.iloc[:, 0].to_frame().values == dtypes_table.values).all( + 1 + ), + index=dtypes_table.index, + ).apply(lambda x: "" if x else "\u2B05") + report << dtypes_table + + with report: + report << tab("Skims Contents", level=3) + ul = xmle.Elem("ul") + for k in skims: + i = xmle.Elem("li") + i << xmle.Elem("b", text=k, tail=f" {skims[k].dtype} {skims[k].dims}") + ul << i + report << ul diff --git a/activitysim/workflows/steps/contrast/directory_prep.py b/activitysim/workflows/steps/contrast/directory_prep.py new file mode 100644 index 0000000000..a7f1245397 --- /dev/null +++ b/activitysim/workflows/steps/contrast/directory_prep.py @@ -0,0 +1,46 @@ +import os +from pathlib import Path + +from ..wrapping import workstep + + +def _prep_dir(directory): + directory = Path(directory) + os.makedirs(directory, exist_ok=True) + os.makedirs(directory / "log", exist_ok=True) + os.makedirs(directory / "trace", exist_ok=True) + gitignore = directory / ".gitignore" + if not os.path.exists(gitignore): + with open(gitignore, "wt") as f: + f.write("/*") + + +@workstep(updates_context=True) +def directory_prep( + tag, + example_name, + workspace, + compile=True, + sharrow=True, + legacy=True, + reference=True, + chunk_training=None, +): + archive_dir = f"{workspace}/{example_name}/output-{tag}" + os.makedirs(archive_dir, exist_ok=True) + if compile: + _prep_dir(f"{archive_dir}/output-compile") + if sharrow: + _prep_dir(f"{archive_dir}/output-sharrow") + if chunk_training: + _prep_dir(f"{archive_dir}/output-sharrow-training") + if legacy: + _prep_dir(f"{archive_dir}/output-legacy") + if chunk_training: + _prep_dir(f"{archive_dir}/output-legacy-training") + if reference: + _prep_dir(f"{archive_dir}/output-reference") + return dict( + archive_dir=archive_dir, + archive_base=os.path.basename(archive_dir), + ) diff --git a/activitysim/workflows/steps/contrast/district_to_district.py b/activitysim/workflows/steps/contrast/district_to_district.py new file mode 100644 index 0000000000..93358ea78d --- /dev/null +++ b/activitysim/workflows/steps/contrast/district_to_district.py @@ -0,0 +1,74 @@ +import logging + +import altair as alt +import pandas as pd +from pypyr.context import Context + +from ....standalone.data_dictionary import check_data_dictionary +from ..progression import reset_progress_step +from ..wrapping import workstep + +logger = logging.getLogger(__name__) + + +@workstep +def compare_district_to_district( + tablesets, + tablename, + district_id, + orig_col="home_zone_id", + dest_col="workplace_zone_id", + orig_label="home_district", + dest_label="work_district", + filter=None, + data_dictionary=None, + size_label="n_workers", + viz_engine="altair", +): + data_dictionary = check_data_dictionary(data_dictionary) + + d = {} + + for key, tableset in tablesets.items(): + if filter is not None: + subtable = tableset[tablename].query(filter) + else: + subtable = tableset[tablename] + district_map = tableset["land_use"][district_id] + orig_district = subtable[orig_col].map(district_map).rename(orig_label) + dest_district = subtable[dest_col].map(district_map).rename(dest_label) + df = subtable.groupby([orig_district, dest_district]).size().rename(size_label) + d[key] = df + + all_d = pd.concat(d, names=["source"]).reset_index() + + district_names = data_dictionary.get("land_use", {}).get(district_id, None) + if district_names is not None: + all_d[orig_label] = all_d[orig_label].map(district_names) + all_d[dest_label] = all_d[dest_label].map(district_names) + + if viz_engine is None: + return all_d + + selection = alt.selection_multi( + fields=[dest_label], + bind="legend", + ) + + fig = ( + alt.Chart(all_d) + .mark_bar() + .encode( + color=f"{dest_label}:N", + y=alt.Y("source", axis=alt.Axis(grid=False, title=""), sort=None), + x=alt.X(size_label, axis=alt.Axis(grid=False)), + row=f"{orig_label}:N", + opacity=alt.condition(selection, alt.value(1), alt.value(0.2)), + tooltip=[f"{orig_label}:N", f"{dest_label}:N", "source", size_label], + ) + .add_selection( + selection, + ) + ) + + return fig diff --git a/activitysim/workflows/steps/contrast/join_table_data.py b/activitysim/workflows/steps/contrast/join_table_data.py new file mode 100644 index 0000000000..528f169d44 --- /dev/null +++ b/activitysim/workflows/steps/contrast/join_table_data.py @@ -0,0 +1,30 @@ +import logging + +from ..progression import reset_progress_step +from ..wrapping import workstep + +logger = logging.getLogger(__name__) + + +@workstep(updates_context=True) +def join_table_data( + tablesets, + tablename, + from_tablename, + columns, + on, +) -> dict: + if isinstance(columns, str): + columns = [columns] + if len(columns) == 1: + columns_note = columns[0] + else: + columns_note = f"{len(columns)} columns" + + reset_progress_step(description=f"join table data / {tablename} <- {columns_note}") + + for key, tableset in tablesets.items(): + other = tableset[from_tablename][columns] + tablesets[key][tablename] = tablesets[key][tablename].join(other, on=on) + + return dict(tablesets=tablesets) diff --git a/activitysim/workflows/steps/contrast/load_skims.py b/activitysim/workflows/steps/contrast/load_skims.py new file mode 100644 index 0000000000..85071b8f75 --- /dev/null +++ b/activitysim/workflows/steps/contrast/load_skims.py @@ -0,0 +1,164 @@ +import glob +import logging +import os +from pathlib import Path + +import numpy as np +import openmatrix +import pandas as pd +import sharrow as sh +import yaml + +from activitysim.standalone.utils import chdir + +from ..wrapping import workstep + +logger = logging.getLogger(__name__) + + +def load_skims_per_settings( + network_los_settings_filename, + data_dir, +): + with open(network_los_settings_filename, "rt") as f: + settings = yaml.safe_load(f) + + skim_settings = settings["taz_skims"] + if isinstance(skim_settings, str): + skims_omx_fileglob = skim_settings + skims_filenames = glob.glob(os.path.join(data_dir, skims_omx_fileglob)) + elif isinstance(skim_settings, list): + skims_filenames = [os.path.join(data_dir, i) for i in skim_settings] + else: + skims_omx_fileglob = skim_settings.get("omx", None) + skims_omx_fileglob = skim_settings.get("files", skims_omx_fileglob) + skims_filenames = glob.glob(os.path.join(data_dir, skims_omx_fileglob)) + index_names = ("otaz", "dtaz", "time_period") + indexes = None + time_period_breaks = settings.get("skim_time_periods", {}).get("periods") + time_periods = settings.get("skim_time_periods", {}).get("labels") + time_period_sep = "__" + + time_window = settings.get("skim_time_periods", {}).get("time_window") + period_minutes = settings.get("skim_time_periods", {}).get("period_minutes") + n_periods = int(time_window / period_minutes) + + tp_map = {} + tp_imap = {} + label = time_periods[0] + i = 0 + for t in range(n_periods): + if t in time_period_breaks: + i = time_period_breaks.index(t) + label = time_periods[i] + tp_map[t + 1] = label + tp_imap[t + 1] = i + + omxs = [ + openmatrix.open_file(skims_filename, mode="r") + for skims_filename in skims_filenames + ] + if isinstance(time_periods, (list, tuple)): + time_periods = np.asarray(time_periods) + result = sh.dataset.from_omx_3d( + omxs, + index_names=index_names, + indexes=indexes, + time_periods=time_periods, + time_period_sep=time_period_sep, + ) + result.attrs["time_period_map"] = tp_map + result.attrs["time_period_imap"] = tp_imap + + # load sparse MAZ skims, if any + from activitysim.core.skim_dataset import load_sparse_maz_skims + + zone_system = int(settings.get("zone_system")) + + if zone_system > 1: + maz2taz_file_name = settings.get("maz") + maz_to_maz = settings.get("maz_to_maz", {}) + maz_to_maz_tables = maz_to_maz.get("tables", ()) + max_blend_distance = maz_to_maz.get("max_blend_distance", {}) + + maz2taz = pd.read_csv(os.path.join(data_dir, maz2taz_file_name), index_col=0) + maz2taz = maz2taz.rename_axis("MAZ") + maz2taz = maz2taz.reset_index() + remapper = dict(zip(maz2taz.MAZ, maz2taz.index)) + + result = load_sparse_maz_skims( + result, + maz2taz.index, + remapper, + zone_system, + maz2taz_file_name, + maz_to_maz_tables=maz_to_maz_tables, + max_blend_distance=max_blend_distance, + data_file_resolver=lambda f, mandatory=True: os.path.join(data_dir, f), + ) + + return result + + +@workstep("skims") +def load_skims( + config_dirs=("configs",), + data_dir="data", + working_directory=None, + common_directory=None, +): + """ + Open and prepare one or more sets of skims for use. + + The xarray library can use dask to delay loading skim matrices until they + are actually needed, so this may not actually load the skims into RAM. + + Parameters + ---------- + config_dirs : Tuple[Path-like] or Mapping[str, Tuple[Path-like]], default ('configs',) + Location of the config directory(s). The skims are loaded by finding + the `network_los.yaml` definition files in one of these directories. + If a mapping is provided, the keys label multiple different sets of skims + to load, which can be useful in contrasting model results from different + input values. + data_dir : Path-like or Mapping[str, Path-like], default 'data' + Loc + working_directory : Path-like, optional + common_directory : Path-like, optional + Deprecated, use working_directory + + Returns + ------- + xarray.Dataset or Dict[str, xarray.Dataset] + A single dataset is returned when a single set of config and data + directories are provided, otherwise a dict is returned with keys + that match the inputs. + """ + if common_directory is not None: + working_directory = common_directory + + if isinstance(config_dirs, str): + config_dirs = [config_dirs] + + if working_directory is None: + working_directory = os.getcwd() + + with chdir(working_directory): + network_los_file = None + for config_dir in config_dirs: + network_los_file = os.path.join(config_dir, "network_los.yaml") + if os.path.exists(network_los_file): + break + if network_los_file is None: + raise FileNotFoundError("<>/network_los.yaml") + if isinstance(data_dir, (str, Path)) and isinstance( + network_los_file, (str, Path) + ): + skims = load_skims_per_settings(network_los_file, data_dir) + else: + skims = {} + for k in data_dir.keys(): + skims[k] = load_skims_per_settings(network_los_file, data_dir[k]) + # TODO: allow for different network_los_file + + return skims diff --git a/activitysim/workflows/steps/contrast/load_tables.py b/activitysim/workflows/steps/contrast/load_tables.py new file mode 100644 index 0000000000..16e33920e8 --- /dev/null +++ b/activitysim/workflows/steps/contrast/load_tables.py @@ -0,0 +1,78 @@ +import os +from pathlib import Path + +from pypyr.context import Context + +from activitysim.standalone.compare import load_final_tables +from activitysim.standalone.utils import chdir + +from ..error_handler import error_logging +from ..progression import reset_progress_step +from ..wrapping import workstep + +# databases = context.get_formatted('databases') +# # the various different output directories to process, for example: +# # { +# # "sharrow": "output-sharrow", +# # "legacy": "output-legacy", +# # } +# +# tables = context.get_formatted('tables') +# # the various tables in the output directories to read, for example: +# # trips: +# # filename: final_trips.csv +# # index_col: trip_id +# # persons: +# # filename: final_persons.csv +# # index_col: person_id +# # land_use: +# # filename: final_land_use.csv +# # index_col: zone_id + + +@workstep("tablesets") +def load_tables(databases, tables, common_output_directory=None) -> dict: + """ + Load tables from one or more tablesets. + + Parameters + ---------- + databases : Dict[str,Path-like] + Defines one or more tablesets to load, out of input or output + directories. Each included database should include all the tables + referenced by the `tables` argument. + tables : Dict[str,Dict[str,str]] + The keys give names of tables to load, and the values are dictionaries + with keys at least including `filename`, and possibly also `index_col`. + common_output_directory : Path-like, optional + The directory in which each of the database directories can be found. + If they are not in the same place, the user should set this to a common + root directory, and include the full relative path for each database. + If not given, defaults to the current working directory. + + Returns + ------- + dict + The loaded tables are under the key 'tablesets'. + """ + + tablefiles = {} + index_cols = {} + for t, v in tables.items(): + if isinstance(v, str): + tablefiles[t] = v + else: + tablefiles[t] = v.get("filename") + index_cols[t] = v.get("index_col", None) + + if common_output_directory is None: + common_output_directory = os.getcwd() + + with chdir(common_output_directory): + tablesets = load_final_tables( + databases, + tablefiles, + index_cols, + ) + + return tablesets diff --git a/activitysim/workflows/steps/contrast/memory_use.py b/activitysim/workflows/steps/contrast/memory_use.py new file mode 100644 index 0000000000..c703188d95 --- /dev/null +++ b/activitysim/workflows/steps/contrast/memory_use.py @@ -0,0 +1,65 @@ +import logging + +import altair as alt +import numpy as np +import pandas as pd + +from ..progression import reset_progress_step +from ..wrapping import workstep + +logger = logging.getLogger(__name__) + + +@workstep +def memory_use( + combined_mem_log, + include_runs=("legacy", "sharrow", "reference"), + relabel_tablesets=None, + memory_measure="uss", +): + reset_progress_step(description="report model memory usage") + + if relabel_tablesets is None: + relabel_tablesets = {} + + include_runs = list(include_runs) + + logger.info(f"building memory use report from {combined_mem_log}") + + df = ( + pd.read_csv( + combined_mem_log, + header=[0, 1], + skipinitialspace=True, + index_col=0, + ) + .fillna(0) + .astype(np.float64) + ) + df.columns = df.columns.set_names(["source", "mem"]) + mem = df.stack().query(f"mem == '{memory_measure}'") + include_runs = [i for i in include_runs if i in mem.columns] + mem = ( + mem[include_runs] + .reset_index() + .reset_index() + .set_index(["event", "index"]) + .drop(columns=["mem"]) + .stack() + .rename("mem_use") + .reset_index() + ) + mem["mem_gigs"] = mem["mem_use"] / 1e9 + mem["source"] = mem["source"].map(lambda x: relabel_tablesets.get(x, x)) + + chart = ( + alt.Chart(mem) + .mark_line() + .encode( + y="mem_gigs", + color="source", + x=alt.X("index", sort=None), + tooltip=["event", "source", "mem_gigs"], + ) + ) + return chart diff --git a/activitysim/workflows/steps/contrast/memory_use_peak.py b/activitysim/workflows/steps/contrast/memory_use_peak.py new file mode 100644 index 0000000000..b6a8b1ed7f --- /dev/null +++ b/activitysim/workflows/steps/contrast/memory_use_peak.py @@ -0,0 +1,65 @@ +import logging + +import altair as alt +import numpy as np +import pandas as pd + +from ..progression import reset_progress_step +from ..wrapping import workstep + +logger = logging.getLogger(__name__) + + +@workstep +def memory_use_peak( + combined_peakmem_log, + include_runs=("legacy", "sharrow", "reference"), + relabel_tablesets=None, + memory_measure="uss", +): + reset_progress_step(description="report component peak memory use") + + if relabel_tablesets is None: + relabel_tablesets = {} + + include_runs = list(include_runs) + + logger.info(f"building peak memory use report from {combined_peakmem_log}") + + try: + df = pd.read_csv( + combined_peakmem_log, + header=[0, 1], + skipinitialspace=True, + index_col=0, + ) + except FileNotFoundError: + return f"File Not Found: {combined_peakmem_log}" + df = df.fillna(0).astype(np.float64) + df.columns = df.columns.set_names(["source", "mem"]) + mem = df.stack().query(f"mem == '{memory_measure}'") + include_runs = [i for i in include_runs if i in mem.columns] + mem = ( + mem[include_runs] + .reset_index() + .reset_index() + .set_index(["event", "index"]) + .drop(columns=["mem"]) + .stack() + .rename("mem_use") + .reset_index() + ) + mem["mem_gigs"] = mem["mem_use"] / 1e9 + mem["source"] = mem["source"].map(lambda x: relabel_tablesets.get(x, x)) + + chart = ( + alt.Chart(mem) + .mark_line(interpolate="step") + .encode( + y="mem_gigs", + color="source", + x=alt.X("event", sort=None), + tooltip=["event", "source", "mem_gigs"], + ) + ) + return chart diff --git a/activitysim/workflows/steps/contrast/nominal_choice.py b/activitysim/workflows/steps/contrast/nominal_choice.py new file mode 100644 index 0000000000..c9756b72b9 --- /dev/null +++ b/activitysim/workflows/steps/contrast/nominal_choice.py @@ -0,0 +1,148 @@ +import logging + +import altair as alt +import pandas as pd +from pypyr.context import Context + +from ..progression import reset_progress_step +from ..wrapping import workstep + +logger = logging.getLogger(__name__) + + +def parse_grouping(g): + if isinstance(g, str): + return g, {"shorthand": g} + elif isinstance(g, dict): + return g.get("field"), g + elif g is None: + return None, None + else: + raise ValueError(g) + + +@workstep +def compare_nominal_choice( + tablesets, + tablename, + nominal_col, + row_grouping=None, + col_grouping=None, + count_label=None, + share_label=None, + axis_label="Share", + title=None, + ordinal=False, + plot_type="share", + relabel_tablesets=None, +): + """ + Parameters + ---------- + tablesets : Mapping + title : str, optional + grouping : str + relabel_tablesets : Mapping[str,str] + Remap the keys in `tablesets` with these values. Any + missing values are retained. This allows you to modify + the figure to e.g. change "reference" to "v1.0.4" without + editing the original input data. + """ + if count_label is None: + count_label = f"# of {tablename}" + if share_label is None: + share_label = f"share of {tablename}" + if relabel_tablesets is None: + relabel_tablesets = {} + + row_g, row_g_kwd = parse_grouping(row_grouping) + col_g, col_g_kwd = parse_grouping(col_grouping) + + d = {} + groupings = [] + if row_g is not None: + groupings.append(row_g) + if col_g is not None: + groupings.append(col_g) + + for key, tableset in tablesets.items(): + df = ( + tableset[tablename] + .groupby(groupings + [nominal_col]) + .size() + .rename(count_label) + .reset_index() + ) + if not groupings: + df[share_label] = df[count_label] / df[count_label].sum() + else: + df[share_label] = df[count_label] / df.groupby(groupings)[ + count_label + ].transform("sum") + d[relabel_tablesets.get(key, key)] = df + + all_d = pd.concat(d, names=["source"]).reset_index() + + selection = alt.selection_multi( + fields=[nominal_col], + bind="legend", + ) + + if plot_type == "count": + x = alt.X( + count_label, + axis=alt.Axis(grid=False, labels=False, title=axis_label), + ) + elif plot_type == "share": + x = alt.X( + share_label, + axis=alt.Axis(grid=False, labels=False, title=axis_label), + scale=alt.Scale(domain=[0.0, 1.0]), + ) + else: + raise ValueError(f"unknown plot_type {plot_type}") + + encode = dict( + color=alt.Color( + nominal_col, + type="ordinal" if ordinal else "nominal", + ), + y=alt.Y("source", axis=alt.Axis(grid=False, title=""), sort=None), + x=x, + opacity=alt.condition(selection, alt.value(1), alt.value(0.2)), + tooltip=[ + nominal_col, + "source", + count_label, + alt.Tooltip(f"{share_label}:Q", format=".2%"), + ], + ) + if row_g is not None: + encode["row"] = alt.Row(**row_g_kwd) + if col_g is not None: + encode["column"] = alt.Column(**col_g_kwd) + + fig = ( + alt.Chart(all_d) + .mark_bar() + .encode( + **encode, + ) + .add_selection( + selection, + ) + ) + + if title: + fig = fig.properties(title=title).configure_title( + fontSize=20, + anchor="start", + color="black", + ) + + if col_grouping is not None: + fig = fig.properties( + width=100, + ) + + return fig diff --git a/activitysim/workflows/steps/contrast/ordinal_distribution.py b/activitysim/workflows/steps/contrast/ordinal_distribution.py new file mode 100644 index 0000000000..a50d25b40b --- /dev/null +++ b/activitysim/workflows/steps/contrast/ordinal_distribution.py @@ -0,0 +1,112 @@ +import logging + +import altair as alt +import pandas as pd +from pypyr.context import Context + +from ..progression import reset_progress_step +from ..wrapping import workstep + +logger = logging.getLogger(__name__) + + +@workstep +def ordinal_distribution( + tablesets, + tablename, + ordinal_col, + facet_grouping="primary_purpose", + title=None, + axis_label=None, + count_label=None, + share_label=None, + interpolate="monotone", + plot_type="share", + value_format=None, +): + if count_label is None: + count_label = f"# of {tablename}" + + if share_label is None: + share_label = f"share of {tablename}" + + if axis_label is None: + axis_label = ordinal_col + groupings = [] + if isinstance(facet_grouping, str): + groupings = [facet_grouping] + + d = {} + for key, tableset in tablesets.items(): + df = ( + tableset[tablename] + .groupby(groupings + [ordinal_col]) + .size() + .rename(count_label) + .unstack(ordinal_col) + .fillna(0) + .stack() + .rename(count_label) + .reset_index() + ) + df[share_label] = df[count_label] / df.groupby(groupings)[ + count_label + ].transform("sum") + d[key] = df + + # This is sorted in reverse alphabetical order by source, so that + # the stroke width for the first line plotted is fattest, and progressively + # thinner lines are plotted over that, so all data is visible on the figure. + all_d = ( + pd.concat(d, names=["source"]) + .reset_index() + .sort_values("source", ascending=False) + ) + + if plot_type == "count": + y = alt.Y(count_label, axis=alt.Axis(grid=False, title=""), stack=None) + elif plot_type == "share": + y = alt.Y(share_label, axis=alt.Axis(grid=False, title=""), stack=None) + else: + raise ValueError(f"unknown plot_type {plot_type}") + + val_format = {} + if value_format is not None: + val_format["format"] = value_format + + fig = ( + alt.Chart(all_d) + .mark_area( + interpolate=interpolate, + fillOpacity=0.3, + line=True, + ) + .encode( + color="source", + y=y, + x=alt.X( + ordinal_col, axis=alt.Axis(grid=False, title=axis_label, **val_format) + ), + tooltip=[ + alt.Tooltip(ordinal_col, **val_format), + "source", + count_label, + alt.Tooltip(f"{share_label}:Q", format=".2%"), + ], + facet=alt.Facet(facet_grouping, columns=3), + strokeWidth="source", + ) + .properties( + width=200, + height=120, + ) + ) + + if title: + fig = fig.properties(title=title).configure_title( + fontSize=20, + anchor="start", + color="black", + ) + + return fig diff --git a/activitysim/workflows/steps/contrast/runtime.py b/activitysim/workflows/steps/contrast/runtime.py new file mode 100644 index 0000000000..e06d9df7a0 --- /dev/null +++ b/activitysim/workflows/steps/contrast/runtime.py @@ -0,0 +1,150 @@ +import logging + +import altair as alt +import pandas as pd + +from ....standalone.utils import chdir +from ..progression import reset_progress_step +from ..wrapping import workstep + +logger = logging.getLogger(__name__) + + +@workstep +def contrast_runtime( + combined_timing_log, + include_runs=( + "legacy", + "sharrow", + ), + relabel_tablesets=None, +): + """ + Generate a figure contrasting runtime for each model component. + + Parameters + ---------- + combined_timing_log + include_runs + relabel_tablesets : Mapping[str,str] + Remap the keys in `include_runs` with these values. Any + missing values are retained. This allows you to modify + the figure to e.g. change "reference" to "v1.0.4" without + editing the original input data. + + Returns + ------- + + """ + reset_progress_step(description="report model runtime") + + if relabel_tablesets is None: + relabel_tablesets = {} + + include_runs = list(include_runs) + + logger.info(f"building runtime report from {combined_timing_log}") + + df = pd.read_csv(combined_timing_log, index_col="model_name") + include_runs = [i for i in include_runs if i in df.columns] + df1 = ( + df[include_runs] + .rename_axis(columns="source") + .unstack() + .rename("seconds") + .reset_index() + .fillna(0) + ) + + def relabel_source(x): + return relabel_tablesets.get(x, x) + + df1["source"] = df1["source"].map(relabel_source) + c = alt.Chart( + df1, + height={"step": 20}, + ) + + if len(include_runs) == 1: + + result = c.mark_bar(size=6,).encode( + x=alt.X("seconds:Q", stack=None), + y=alt.Y("model_name", type="nominal", sort=None), + color="source", + tooltip=["source", "model_name", "seconds"], + ) | alt.Chart(df1).mark_bar().encode( + color="source", + x="source", + y="sum(seconds)", + tooltip=["source", "sum(seconds)"], + ) + + elif len(include_runs) == 2: + + result = c.mark_bar(yOffset=-3, size=6,).transform_filter( + (alt.datum.source == relabel_source(include_runs[0])) + ).encode( + x=alt.X("seconds:Q", stack=None), + y=alt.Y("model_name", type="nominal", sort=None), + color="source", + tooltip=["source", "model_name", "seconds"], + ) + c.mark_bar( + yOffset=4, + size=6, + ).transform_filter( + (alt.datum.source == relabel_source(include_runs[1])) + ).encode( + x=alt.X("seconds:Q", stack=None), + y=alt.Y("model_name", type="nominal", sort=None), + color="source", + tooltip=["source", "model_name", "seconds"], + ) | alt.Chart( + df1 + ).mark_bar().encode( + color="source", + x="source", + y="sum(seconds)", + tooltip=["source", "sum(seconds)"], + ) + + elif len(include_runs) == 3: + result = c.mark_bar(yOffset=-5, size=4,).transform_filter( + (alt.datum.source == relabel_source(include_runs[0])) + ).encode( + x=alt.X("seconds:Q", stack=None), + y=alt.Y("model_name", type="nominal", sort=None), + color="source", + tooltip=["source", "model_name", "seconds"], + ) + c.mark_bar( + yOffset=0, + size=4, + ).transform_filter( + (alt.datum.source == relabel_source(include_runs[1])) + ).encode( + x=alt.X("seconds:Q", stack=None), + y=alt.Y("model_name", type="nominal", sort=None), + color="source", + tooltip=["source", "model_name", "seconds"], + ) + c.mark_bar( + yOffset=5, + size=4, + ).transform_filter( + (alt.datum.source == relabel_source(include_runs[2])) + ).encode( + x=alt.X("seconds:Q", stack=None), + y=alt.Y("model_name", type="nominal", sort=None), + color="source", + tooltip=["source", "model_name", "seconds"], + ) | alt.Chart( + df1 + ).mark_bar().encode( + color="source", + x=alt.X("source", type="nominal", sort=None), + y="sum(seconds)", + tooltip=["source", "sum(seconds)"], + ) + + else: + raise ValueError(f"len(include_runs) == {len(include_runs)}") + + return result diff --git a/activitysim/workflows/steps/contrast/transform_data.py b/activitysim/workflows/steps/contrast/transform_data.py new file mode 100644 index 0000000000..1b621631ec --- /dev/null +++ b/activitysim/workflows/steps/contrast/transform_data.py @@ -0,0 +1,72 @@ +import logging + +import numpy as np +import pandas as pd +from pypyr.context import Context + +from ..progression import reset_progress_step +from ..wrapping import workstep + +logger = logging.getLogger(__name__) + + +@workstep(updates_context=True) +def transform_data( + tablesets, + tablename, + column, + out, + qcut=None, + cut=None, + clip=None, + censor=None, + eval=None, +) -> dict: + + if qcut is None and cut is None and clip is None and censor is None: + raise ValueError("must give at least one of {cut, qcut, clip, censor}") + + reset_progress_step(description=f"attach transformed data / {tablename} <- {out}") + + if eval is not None: + for key, tableset in tablesets.items(): + for target, expr in eval.items(): + tableset[tablename][target] = tableset[tablename].eval(expr) + return dict(tablesets=tablesets) + + # collect all series into a common vector, so bins are common + pieces = {} + for key, tableset in tablesets.items(): + pieces[key] = tableset[tablename][column] + + common = pd.concat(pieces, names=["source"]) + + if clip is not None: + common = common.clip(**clip) + + if censor is not None: + common = common.where(common.between(**censor)) + + use_midpoint = False + if cut is not None: + if cut.get("labels", None) == "midpoint": + use_midpoint = True + cut.pop("labels") + common = pd.cut(common, **cut) + elif qcut is not None: + if qcut.get("labels", None) == "midpoint": + use_midpoint = True + qcut.pop("labels") + common = pd.qcut(common, **qcut) + + if use_midpoint: + common = common.apply(lambda x: x.mid) + + pieces = {k: common.loc[k] for k in tablesets.keys()} + + for key in tablesets: + tablesets[key][tablename] = tablesets[key][tablename].assign( + **{out: pieces[key]} + ) + + return dict(tablesets=tablesets) diff --git a/activitysim/workflows/steps/contrast/trip_distance.py b/activitysim/workflows/steps/contrast/trip_distance.py new file mode 100644 index 0000000000..d9bb106bab --- /dev/null +++ b/activitysim/workflows/steps/contrast/trip_distance.py @@ -0,0 +1,120 @@ +import logging + +import altair as alt +import pandas as pd + +from ..wrapping import workstep + +logger = logging.getLogger(__name__) + + +@workstep +def trip_distance( + tablesets, + skims, + dist_skim_name, + dist_bins=20, + grouping="primary_purpose", + title=None, + max_dist=None, + relabel_tablesets=None, + tablename="trips", +): + """ + + Parameters + ---------- + tablesets + skims + dist_skim_name + dist_bins + grouping + title + max_dist + relabel_tablesets : Mapping[str,str] + Remap the keys in `tablesets` with these values. Any + missing values are retained. This allows you to modify + the figure to e.g. change "reference" to "v1.0.4" without + editing the original input data. + + Returns + ------- + altair.Chart + """ + if relabel_tablesets is None: + relabel_tablesets = {} + + groupings = [grouping] + if not isinstance(skims, dict): + skims = {i: skims for i in tablesets.keys()} + + distances = {} + for key, tableset in tablesets.items(): + distances[key] = tableset[tablename][dist_skim_name] + + if dist_bins is not None: + result = pd.concat(distances, names=["source"]) + if max_dist is not None: + result = result[result <= max_dist] + result = pd.cut(result, dist_bins).to_frame() + distances = {k: result.loc[k] for k in tablesets.keys()} + + data = {} + for key, tableset in tablesets.items(): + data[key] = tableset[tablename].assign(**{"distance": distances[key]}) + + d = {} + for key, dat in data.items(): + df = ( + dat.groupby(groupings + ["distance"]) + .size() + .rename("n_trips") + .unstack("distance") + .fillna(0) + .stack() + .rename("n_trips") + .reset_index() + ) + df["share_trips"] = df["n_trips"] / df.groupby(groupings)["n_trips"].transform( + "sum" + ) + d[relabel_tablesets.get(key, key)] = df + + # This is sorted in reverse alphabetical order by source, so that + # the stroke width for the first line plotted is fattest, and progressively + # thinner lines are plotted over that, so all data is visible on the figure. + all_d = ( + pd.concat(d, names=["source"]) + .reset_index() + .sort_values("source", ascending=False) + ) + all_d["distance"] = all_d["distance"].apply(lambda x: x.mid) + + fig = ( + alt.Chart(all_d) + .mark_line( + interpolate="monotone", + ) + .encode( + color="source", + y=alt.Y("share_trips", axis=alt.Axis(grid=False, title="")), + x=alt.X("distance", axis=alt.Axis(grid=False, title="Distance")), + # opacity=alt.condition(selection, alt.value(1), alt.value(0.2)), + # tooltip = ['trip_mode', 'source', 'n_trips', alt.Tooltip('share_trips:Q', format='.2%')], + facet=alt.Facet(grouping, columns=3), + strokeWidth="source", + ) + .properties( + width=200, + height=120, + ) + ) + + if title: + fig = fig.properties(title=title).configure_title( + fontSize=20, + anchor="start", + color="black", + ) + + return fig diff --git a/activitysim/workflows/steps/copy_files.py b/activitysim/workflows/steps/copy_files.py new file mode 100644 index 0000000000..e9ee247af9 --- /dev/null +++ b/activitysim/workflows/steps/copy_files.py @@ -0,0 +1,34 @@ +import glob +import os +import shutil + +from pypyr.errors import KeyNotInContextError +from pypyr.steps.fetchyaml import run_step as _fetch +from pypyr.steps.filewriteyaml import run_step as _write +from pypyr.steps.py import run_step as _run_step + +from .progression import progress, progress_overall, progress_step +from .wrapping import workstep + + +@workstep +def copy_files( + source_glob, + dest_dir, +) -> None: + """ + Copy files. + + Parameters + ---------- + source_glob : str or Sequence[str] + One or more file glob patterns to copy + dest_dir : path-like + Files that match the source_glob(s) will be copied here. + """ + os.makedirs(dest_dir, exist_ok=True) + if isinstance(source_glob, str): + source_glob = [source_glob] + for pattern in source_glob: + for filename in glob.glob(pattern): + shutil.copy2(filename, dest_dir) diff --git a/activitysim/workflows/steps/create.py b/activitysim/workflows/steps/create.py new file mode 100644 index 0000000000..70857240c7 --- /dev/null +++ b/activitysim/workflows/steps/create.py @@ -0,0 +1,53 @@ +import os +import shlex + +from ...standalone.utils import chdir +from .progression import reset_progress_step +from .wrapping import workstep + + +@workstep +def create(example_name, destination=None) -> None: + """ + Install a functioning example from the ActivitySim resources collection. + + The behavior of this workstep differs slightly from the CLI of + `create`: on the command line, if the destination directory does not + already exist, it is created and becomes the target location to install + the example model. This can result in the example model being installed + into a (newly created) directory with any name, not necessarily the name + of the example being installed. + + This workstep function creates the destination directory if it does not + exist, but then always creates a subdirectory within the destination, named + according to the `example_name`. This ensures stability and guarantees the + resulting installed example is in the same location whether that directory + existed before or not. + + Parameters + ---------- + example_name : str + The name of the example to be installed, which should be listed in + the example manifest. + destination : path-like, optional + The directory where the example should be installed, defaulting to the + current working directory. The example is always then installed into + "destination/example_name". + + """ + reset_progress_step( + description=f"activitysim create {example_name}", prefix="[bold green]" + ) + + args = f"create -e {example_name} --link -d ." + if destination: + os.makedirs(destination, exist_ok=True) + else: + destination = "." + + # Call the run program inside this process + from activitysim.cli.main import prog + + with chdir(destination): + namespace = prog().parser.parse_args(shlex.split(args)) + namespace.afunc(namespace) diff --git a/activitysim/workflows/steps/error_handler.py b/activitysim/workflows/steps/error_handler.py new file mode 100644 index 0000000000..97e85bb618 --- /dev/null +++ b/activitysim/workflows/steps/error_handler.py @@ -0,0 +1,14 @@ +import logging + + +def error_logging(func): + def wrapper(*args, **kwargs): + try: + return func(*args, **kwargs) + except Exception as err: + logging.error(f"===== ERROR IN {func.__name__} =====") + logging.exception(f"{err}") + logging.error(f"===== / =====") + raise + + return wrapper diff --git a/activitysim/workflows/steps/install_env.py b/activitysim/workflows/steps/install_env.py new file mode 100644 index 0000000000..0a30d53c88 --- /dev/null +++ b/activitysim/workflows/steps/install_env.py @@ -0,0 +1,60 @@ +import logging +import os.path +import subprocess + +from .cmd.dsl import stream_process +from .progression import reset_progress_step +from .wrapping import workstep + +logger = logging.getLogger(__name__) + + +@workstep(returns_names="install_env_returncode") +def install_env( + env_prefix, + asim_version="1.0.4", + cwd=None, + label=None, +): + if os.path.exists(env_prefix): + return 0 + reset_progress_step(description=f"Creating activitysim v{asim_version} environment") + + os.makedirs(os.path.dirname(env_prefix), exist_ok=True) + command = [ + "mamba", + "create", + "--prefix", + env_prefix, + f"python=3.9", + f"activitysim={asim_version}", + "-c", + "conda-forge", + "--override-channels", + "--yes", + ] + + if label is None: + label = f"Creating {asim_version} Environment" + + logger.info(f"running command:\n{' '.join(command)}") + process = subprocess.Popen( + " ".join(command), + shell=True, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + cwd=cwd, + ) + stream_process(process, label) + + # don't swallow the error, because it's the Step swallow decorator + # responsibility to decide to ignore or not. + if process.returncode: + raise subprocess.CalledProcessError( + process.returncode, + process.args, + process.stdout, + process.stderr, + ) + + return process.returncode diff --git a/activitysim/workflows/steps/main.py b/activitysim/workflows/steps/main.py new file mode 100644 index 0000000000..80de957e74 --- /dev/null +++ b/activitysim/workflows/steps/main.py @@ -0,0 +1,103 @@ +"""Naive custom loader without any error handling.""" +import contextlib +import os +import signal +import sys +import traceback +from pathlib import Path + +from .progression import get_progress + + +def get_pipeline_definition(pipeline_name, parent): + """Simplified loader that gets pipeline_name.yaml in working dir.""" + import pypyr.yaml + from pypyr.loaders.file import get_pipeline_definition + + search_dir = Path(os.path.dirname(__file__)).parent + workflow_file = search_dir.joinpath(f"{pipeline_name}.yaml") + if os.path.exists(workflow_file): + with open(workflow_file) as yaml_file: + return pypyr.yaml.get_pipeline_yaml(yaml_file) + else: + + return get_pipeline_definition(pipeline_name, parent) + + +def enable_vt_support(): + # allow printing in color on windows terminal + if os.name == "nt": + import ctypes + + hOut = ctypes.windll.kernel32.GetStdHandle(-11) + out_modes = ctypes.c_uint32() + ENABLE_VT_PROCESSING = ctypes.c_uint32(0x0004) + ctypes.windll.kernel32.GetConsoleMode(hOut, ctypes.byref(out_modes)) + out_modes = ctypes.c_uint32(out_modes.value | 0x0004) + ctypes.windll.kernel32.SetConsoleMode(hOut, out_modes) + + +def main(args): + """ + Run a named workflow. + + Each workflow defines its own arguments, refer to the workflow itself to + learn what the arguments and options are. + """ + if args is None: + args = sys.argv[2:] + + if "--no-rich" in args: + args.remove("--no-rich") + os.environ["NO_RICH"] = "1" + + with get_progress(): + + try: + import pypyr.log.logger + import pypyr.pipelinerunner + import pypyr.yaml + from pypyr.cli import get_args + from pypyr.config import config + except ImportError: + raise ImportError("activitysim.workflows requires pypyr") + + parsed_args = get_args(args) + + try: + config.init() + pypyr.log.logger.set_root_logger( + log_level=parsed_args.log_level, log_path=parsed_args.log_path + ) + + pypyr.pipelinerunner.run( + pipeline_name=parsed_args.pipeline_name, + args_in=parsed_args.context_args, + parse_args=True, + groups=parsed_args.groups, + success_group=parsed_args.success_group, + failure_group=parsed_args.failure_group, + py_dir=parsed_args.py_dir, + loader="activitysim.workflows", + ) + + except KeyboardInterrupt: + # Shell standard is 128 + signum = 130 (SIGINT = 2) + sys.stdout.write("\n") + return 128 + signal.SIGINT + except Exception as e: + # stderr and exit code 255 + sys.stderr.write("\n") + sys.stderr.write(f"\033[91m{type(e).__name__}: {str(e)}\033[0;0m") + sys.stderr.write("\n") + # at this point, you're guaranteed to have args and thus log_level + if parsed_args.log_level: + if parsed_args.log_level < 10: + # traceback prints to stderr by default + traceback.print_exc() + + # if we are in the debugger, re-raise the error instead of returning + if sys.gettrace() is not None: + raise + + return 255 diff --git a/activitysim/workflows/steps/make_tag.py b/activitysim/workflows/steps/make_tag.py new file mode 100644 index 0000000000..59a8a5e3b5 --- /dev/null +++ b/activitysim/workflows/steps/make_tag.py @@ -0,0 +1,47 @@ +import time + +from .progression import reset_progress_step +from .wrapping import workstep + + +@workstep(updates_context=True) +def make_tag( + example_name, + tag=None, + compile=True, + sharrow=True, + legacy=True, + resume_after=True, + fast=True, + mp=False, + reference=None, + reference_asim_version="0.0.0", + multiprocess=0, + chunk_training_mode=None, + main_n_households=None, +): + reset_progress_step(description="Initialize Tag") + if tag is None: + tag = time.strftime("%Y-%m-%d-%H%M%S") + contrast = sharrow and legacy + + flags = [] + if resume_after: + flags.append(f" -r {resume_after}") + if fast: + flags.append("--fast") + + out = dict(tag=tag, contrast=contrast, flags=" ".join(flags)) + if isinstance(reference, str) and "." in reference: + out["reference_asim_version"] = reference + out["reference"] = True + out["relabel_tablesets"] = {"reference": f"v{reference_asim_version}"} + out["is_multiprocess"] = multiprocess > 1 + out["num_processes"] = int(multiprocess) + + if chunk_training_mode and chunk_training_mode != "disabled": + out["chunk_application_mode"] = "production" + else: + out["chunk_application_mode"] = chunk_training_mode + + return out diff --git a/activitysim/workflows/steps/memory_stress_test.py b/activitysim/workflows/steps/memory_stress_test.py new file mode 100644 index 0000000000..0fc42d7c20 --- /dev/null +++ b/activitysim/workflows/steps/memory_stress_test.py @@ -0,0 +1,35 @@ +import logging +import os +import time + +import numpy as np +import psutil + +from ...core.util import si_units +from .wrapping import workstep + + +def ping_mem(pid=None): + if pid is None: + pid = os.getpid() + current_process = psutil.Process(pid) + with current_process.oneshot(): + info = current_process.memory_full_info() + uss = info.uss + rss = info.rss + + return f"USS={si_units(uss)} RSS={si_units(rss)}" + + +@workstep(updates_context=True) +def memory_stress_test(n=37): + + logging.critical(f"ping_mem = {ping_mem()}") + big = np.arange(int(2 ** float(n) / 8), dtype=np.float64) + big *= 2.0 + time.sleep(1.0) + logging.critical(f"ping_mem = {ping_mem()}") + time.sleep(5.0) + logging.critical(f"ping_mem = {ping_mem()}") + logging.critical(f"bye") + return {} diff --git a/activitysim/workflows/steps/progression.py b/activitysim/workflows/steps/progression.py new file mode 100644 index 0000000000..50c128ac26 --- /dev/null +++ b/activitysim/workflows/steps/progression.py @@ -0,0 +1,104 @@ +import os +import time +from datetime import timedelta + + +class DummyProgress: + def __init__(self, *args, **kwargs): + pass + + def __enter__(self): + pass + + def __exit__(self, exc_type, exc_val, exc_tb): + pass + + def add_task(self, *args, **kwargs): + pass + + def reset(self, *args, **kwargs): + pass + + def update(self, *args, **kwargs): + pass + + +dummy_progress = DummyProgress() + +try: + + from rich.panel import Panel + from rich.progress import Progress, ProgressColumn, Text + +except ImportError: + + # allow printing in color on windows terminal + if os.name == "nt": + import ctypes + + hOut = ctypes.windll.kernel32.GetStdHandle(-11) + out_modes = ctypes.c_uint32() + ENABLE_VT_PROCESSING = ctypes.c_uint32(0x0004) + ctypes.windll.kernel32.GetConsoleMode(hOut, ctypes.byref(out_modes)) + out_modes = ctypes.c_uint32(out_modes.value | 0x0004) + ctypes.windll.kernel32.SetConsoleMode(hOut, out_modes) + + Progress = DummyProgress + progress_overall = None + progress = dummy_progress + +else: + + class MyProgress(Progress): + def get_renderables(self): + yield Panel(self.make_tasks_table(self.tasks)) + + class TimeElapsedColumn(ProgressColumn): + """Renders time elapsed.""" + + def render(self, task: "Task") -> Text: + """Show time remaining.""" + elapsed = task.finished_time if task.finished else task.elapsed + if elapsed is None: + return Text("-:--:--", style="progress.elapsed") + delta = timedelta(seconds=(elapsed)) + return Text(str(delta)[:-5], style="progress.elapsed") + + progress = MyProgress( + TimeElapsedColumn(), + "[progress.description]{task.description}", + ) + + progress_step = progress.add_task("step") + progress_overall = progress.add_task("overall") + + +def get_progress(): + if os.environ.get("NO_RICH", False): + return dummy_progress + else: + return progress + + +def update_progress_overall(description, formatting=""): + if progress_overall is not None: + if formatting: + if not formatting.startswith("[") and not formatting.endswith("]"): + formatting = f"[{formatting}]" + progress.update(progress_overall, description=f"{formatting}{description}") + else: + progress.update(progress_overall, description=f"{description}") + else: + print("╔" + "═" * (len(description) + 2) + "╗") + print(f"║ {description} ║ {time.strftime('%I:%M:%S %p')}") + print("╚" + "═" * (len(description) + 2) + "╝") + + +def reset_progress_step(*args, description="", prefix="", **kwargs): + if not os.environ.get("NO_RICH", False): + print(f"\u23F1 {time.strftime('%I:%M:%S %p')} - {description}") + progress.reset(progress_step, *args, description=prefix + description, **kwargs) + else: + print("╭" + "─" * (len(description) + 2) + "╮") + print(f"│ {description} │ {time.strftime('%I:%M:%S %p')}") + print("╰" + "─" * (len(description) + 2) + "╯") diff --git a/activitysim/workflows/steps/py.py b/activitysim/workflows/steps/py.py new file mode 100644 index 0000000000..83759d61f9 --- /dev/null +++ b/activitysim/workflows/steps/py.py @@ -0,0 +1,27 @@ +from pypyr.errors import KeyNotInContextError +from pypyr.steps.py import run_step as _run_step + +from .progression import reset_progress_step + + +def run_step(context): + """Execute dynamic python code. + + Takes two forms of input: + py: exec contents as dynamically interpreted python statements, with + contents of context available as vars. + pycode: exec contents as dynamically interpreted python statements, + with the context object itself available as a var. + + Args: + context (pypyr.context.Context): Mandatory. + Context is a dictionary or dictionary-like. + Context must contain key 'py' or 'pycode' + """ + try: + label = context.get_formatted("label") + except KeyNotInContextError: + label = None + if label is not None: + reset_progress_step(description=label) + _run_step(context) diff --git a/activitysim/workflows/steps/pype.py b/activitysim/workflows/steps/pype.py new file mode 100644 index 0000000000..ffd2639190 --- /dev/null +++ b/activitysim/workflows/steps/pype.py @@ -0,0 +1,10 @@ +from pypyr.steps.pype import run_step as _run_step + + +def run_step(context): + pype = context.get("pype") + if isinstance(pype, str): + pype = {"name": pype} + pype["loader"] = "activitysim.workflows" + context["pype"] = pype + return _run_step(context) diff --git a/activitysim/workflows/steps/reporting/__init__.py b/activitysim/workflows/steps/reporting/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/activitysim/workflows/steps/reporting/init_report.py b/activitysim/workflows/steps/reporting/init_report.py new file mode 100644 index 0000000000..3191b99874 --- /dev/null +++ b/activitysim/workflows/steps/reporting/init_report.py @@ -0,0 +1,16 @@ +from xmle import NumberedCaption, Reporter + +from ..progression import reset_progress_step +from ..wrapping import workstep + + +@workstep(updates_context=True) +def init_report( + title, +): + reset_progress_step(description="initialize report") + return dict( + report=Reporter(title=title), + fig=NumberedCaption("Figure", level=2, anchor=True), + tab=NumberedCaption("Table", level=2, anchor=True), + ) diff --git a/activitysim/workflows/steps/reporting/load_data_dictionary.py b/activitysim/workflows/steps/reporting/load_data_dictionary.py new file mode 100644 index 0000000000..80d4aebf5f --- /dev/null +++ b/activitysim/workflows/steps/reporting/load_data_dictionary.py @@ -0,0 +1,23 @@ +import os + +import yaml + +from ..wrapping import workstep + + +@workstep("data_dictionary") +def load_data_dictionary( + config_dirs, + data_dict_filename="data_dictionary.yaml", + cwd=".", +): + if isinstance(config_dirs, str): + config_dirs = [config_dirs] + dd = {} + for config_dir in config_dirs: + if os.path.isdir(os.path.join(cwd, config_dir)): + f = os.path.join(cwd, config_dir, data_dict_filename) + if os.path.exists(f): + with open(f, "rt") as stream: + dd.update(yaml.safe_load(stream)) + return dd diff --git a/activitysim/workflows/steps/reporting/machine_info.py b/activitysim/workflows/steps/reporting/machine_info.py new file mode 100644 index 0000000000..6edbdacafa --- /dev/null +++ b/activitysim/workflows/steps/reporting/machine_info.py @@ -0,0 +1,55 @@ +import multiprocessing +import platform +import subprocess + +import pandas as pd +import psutil + +from ..wrapping import workstep + + +def get_processor_info(): + out = "" + if platform.system() == "Windows": + out = platform.processor() + elif platform.system() == "Darwin": + out = subprocess.check_output( + ["/usr/sbin/sysctl", "-n", "machdep.cpu.brand_string"] + ).strip() + elif platform.system() == "Linux": + command = "cat /proc/cpuinfo" + out = subprocess.check_output(command, shell=True).strip() + if isinstance(out, bytes): + out = out.decode("utf-8") + return out + + +def get_size(bytes, suffix="B"): + """ + Scale bytes to its proper format + """ + factor = 1024 + for unit in ["", "K", "M", "G", "T"]: + if bytes < factor: + return f"{bytes:.2f}{unit}{suffix}" + bytes /= factor + unit = "P" + return f"{bytes:.2f}{unit}{suffix}" + + +@workstep +def machine_info(name=None): + uname = platform.uname() + ram = get_size(psutil.virtual_memory().total) + out = { + "Processor": get_processor_info(), + "Number of Cores": multiprocessing.cpu_count(), + "OS": uname.system, + "Release": uname.release, + "RAM": ram, + } + if name is True: + name = uname.node + if name is None: + name = "" + return pd.DataFrame(pd.Series(out).rename(name)) diff --git a/activitysim/workflows/steps/reporting/save_report.py b/activitysim/workflows/steps/reporting/save_report.py new file mode 100644 index 0000000000..2de2deced9 --- /dev/null +++ b/activitysim/workflows/steps/reporting/save_report.py @@ -0,0 +1,122 @@ +import logging +import os + +from .... import __version__ +from ..wrapping import workstep + +# Get decorated version when in development +try: + top_package = __import__(__name__.split(".")[0]) + import setuptools_scm + + version = setuptools_scm.get_version(os.path.dirname(top_package.__path__[0])) +except: # noqa: E722 + version = __version__ + + +@workstep +def save_report( + report, + html_filename, + toc_color="forest", +): + + bootstrap_font_family = ( + 'font-family: -apple-system, "system-ui", "Segoe UI",' + ' "Helvetica Neue", Arial, sans-serif, "Apple Color Emoji",' + ' "Segoe UI Emoji", "Segoe UI Symbol";' + ) + toc_width = 200 + signature_font = "font-size:70%; font-weight:200;" + signature_name_font = "font-weight:400; font-style:normal;" + + css = ( + """ + html { """ + + bootstrap_font_family + + """ } + div.xmle_title { font-weight:700; font-size: 2em; color: rgb(35, 144, 98);} + body { margin-left: """ + + str(toc_width) + + """px; } + div.xmle_html_report { padding-left: 5px; } + .table_of_contents_frame { width: """ + + str(toc_width - 13) + + """px; position: fixed; margin-left: -""" + + str(toc_width) + + """px; top:0; padding-top:10px; z-index:2000;} + .table_of_contents { width: """ + + str(toc_width - 13) + + """px; position: fixed; margin-left: -""" + + str(toc_width) + + """px; font-size:85%;} + .table_of_contents_head { font-weight:700; padding-left:25px; } + .table_of_contents ul { padding-left:25px; } + .table_of_contents ul ul { font-size:75%; padding-left:15px; } + .xmle_signature {""" + + signature_font + + """ width: """ + + str(toc_width - 30) + + """px; position: fixed; left: 0px; bottom: 0px; padding-left:20px;""" + + """padding-bottom:2px; background-color:rgba(255,255,255,0.9);} + .xmle_name_signature {""" + + signature_name_font + + """} + a.parameter_reference {font-style: italic; text-decoration: none} + .strut2 {min-width:2in} + .histogram_cell { padding-top:1; padding-bottom:1; vertical-align:center; } + table { border-spacing: 0; border: none; } + .dataframe tbody tr:nth-child(odd) { + background: #f5f5f5; + } + .dataframe tr, .dataframe th, .dataframe td { + text-align: right; + vertical-align: middle; + padding: 0.2em 0.2em; + line-height: normal; + white-space: normal; + max-width: none; + border: none; + } + .dataframe table { + margin-left: auto; + margin-right: auto; + border: none; + border-collapse: collapse; + border-spacing: 0; + color: black; + font-size: 12px; + table-layout: fixed; + } + table.floatinghead thead {background-color:#FFF;} + table.dataframe thead {background-color:#FFF;} + @media print { + body { color: #000; background: #fff; width: 100%; margin: 0; padding: 0;} + /*.table_of_contents { display: none; }*/ + @page { + margin: 1in; + } + h1, h2, h3 { page-break-after: avoid; } + img { max-width: 100% !important; } + ul, img, table { page-break-inside: avoid; } + .xmle_signature {""" + + signature_font + + """ padding:0; background-color:#fff; position: fixed; bottom: 0;} + .xmle_name_signature {""" + + signature_name_font + + """} + .xmle_signature img {display:none;} + .xmle_signature .noprint {display:none;} + } + """ + ) + + logging.critical(f"SAVING REPORT TO {(html_filename)}") + report.save( + html_filename, + overwrite=True, + css=css, + toc_font=bootstrap_font_family, + toc_color=toc_color, + branding=f"ActivitySim {version.replace('+', ' +')}", + ) diff --git a/activitysim/workflows/steps/reporting/section_title.py b/activitysim/workflows/steps/reporting/section_title.py new file mode 100644 index 0000000000..edf209907b --- /dev/null +++ b/activitysim/workflows/steps/reporting/section_title.py @@ -0,0 +1,17 @@ +import logging + +from ..wrapping import workstep + +logger = logging.getLogger(__name__) + + +@workstep(returns_names="report") +def section_title( + report, + title, + level=2, +): + with report: + t = "#" * level + report << f"{t} {title}" + return report diff --git a/activitysim/workflows/steps/reporting/settings.py b/activitysim/workflows/steps/reporting/settings.py new file mode 100644 index 0000000000..f2ff77b191 --- /dev/null +++ b/activitysim/workflows/steps/reporting/settings.py @@ -0,0 +1,18 @@ +import multiprocessing +import platform +import subprocess + +import pandas as pd +import psutil + +from ..wrapping import workstep + + +@workstep +def settings(names, **kwargs): + if isinstance(names, str): + names = [names] + out = {} + for name in names: + out[name] = kwargs.get(name, None) + return pd.DataFrame(pd.Series(out).rename("")) diff --git a/activitysim/workflows/steps/run.py b/activitysim/workflows/steps/run.py new file mode 100644 index 0000000000..bce4d1d303 --- /dev/null +++ b/activitysim/workflows/steps/run.py @@ -0,0 +1,79 @@ +import shlex + +from pypyr.errors import KeyNotInContextError + +from ...standalone.utils import chdir +from .progression import reset_progress_step +from .wrapping import workstep + + +def _get_formatted(context, key, default): + try: + out = context.get_formatted(key) + except KeyNotInContextError: + out = None + if out is None: + out = default + return out + + +@workstep +def run_activitysim( + label=None, + cwd=None, + pre_config_dirs=(), + config_dirs=("configs",), + data_dir="data", + output_dir="output", + ext_dirs=None, + resume_after=None, + fast=True, + settings_file=None, +) -> None: + if isinstance(pre_config_dirs, str): + pre_config_dirs = [pre_config_dirs] + else: + pre_config_dirs = list(pre_config_dirs) + if isinstance(config_dirs, str): + config_dirs = [config_dirs] + else: + config_dirs = list(config_dirs) + if isinstance(ext_dirs, str): + ext_dirs = [ext_dirs] + elif ext_dirs is None: + ext_dirs = [] + else: + ext_dirs = list(ext_dirs) + flags = [] + if resume_after: + flags.append(f" -r {resume_after}") + if fast: + flags.append("--fast") + if settings_file: + flags.append(f" -s {settings_file}") + flags = " ".join(flags) + cfgs = " ".join(f"-c {c}" for c in pre_config_dirs + config_dirs) + exts = "".join(f" -e {e}" for e in ext_dirs) + args = f"run {cfgs}{exts} -d {data_dir} -o {output_dir} {flags}" + if label is None: + label = f"activitysim {args}" + + reset_progress_step(description=f"{label}", prefix="[bold green]") + + # Clear all saved state from ORCA + import orca + + orca.clear_cache() + orca.clear_all() + + # Re-inject everything from ActivitySim + from ...core.inject import reinject_decorated_tables + + reinject_decorated_tables(steps=True) + + # Call the run program inside this process + from activitysim.cli.main import prog + + with chdir(cwd): + namespace = prog().parser.parse_args(shlex.split(args)) + namespace.afunc(namespace) diff --git a/activitysim/workflows/steps/run_subprocess.py b/activitysim/workflows/steps/run_subprocess.py new file mode 100644 index 0000000000..35db463475 --- /dev/null +++ b/activitysim/workflows/steps/run_subprocess.py @@ -0,0 +1,161 @@ +import logging +import os +import subprocess +from tempfile import TemporaryFile +from time import sleep + +from pypyr.errors import KeyNotInContextError + +from .progression import reset_progress_step +from .wrapping import workstep + + +def _get_formatted(context, key, default): + try: + out = context.get_formatted(key) + except KeyNotInContextError: + out = None + if out is None: + out = default + return out + + +def _stream_subprocess(args, cwd, env): + with TemporaryFile() as outputstream: + process = subprocess.Popen( + args=args, + shell=True, + stdout=outputstream, + stderr=subprocess.STDOUT, + cwd=cwd, + env=env, + ) + while process.poll() is None: + where = outputstream.tell() + lines = outputstream.read() + if not lines: + # Adjust the sleep interval to your needs + sleep(0.25) + # make sure pointing to the last place we read + outputstream.seek(where) + else: + # Windows adds an extra carriage return and then chokes on + # it when displaying (or, as it were, not displaying) the + # output. So we give Windows a little helping hand. + print(lines.decode().replace("\r\n", "\n"), end="") + return process + + +@workstep +def run_activitysim_as_subprocess( + label=None, + cwd=None, + pre_config_dirs=(), + config_dirs=("configs",), + data_dir="data", + output_dir="output", + ext_dirs=None, + settings_file=None, + resume_after=None, + fast=True, + conda_prefix=None, + single_thread=True, + multi_thread=None, +) -> None: + if isinstance(pre_config_dirs, str): + pre_config_dirs = [pre_config_dirs] + else: + pre_config_dirs = list(pre_config_dirs) + if isinstance(config_dirs, str): + config_dirs = [config_dirs] + else: + config_dirs = list(config_dirs) + if isinstance(ext_dirs, str): + ext_dirs = [ext_dirs] + elif ext_dirs is None: + ext_dirs = [] + else: + ext_dirs = list(ext_dirs) + flags = [] + if resume_after: + flags.append(f" -r {resume_after}") + if fast: + flags.append("--fast") + if settings_file: + flags.append(f"-s {settings_file}") + flags = " ".join(flags) + cfgs = " ".join(f"-c {c}" for c in pre_config_dirs + config_dirs) + exts = "".join(f" -e {e}" for e in ext_dirs) + args = f"activitysim run {cfgs}{exts} -d {data_dir} -o {output_dir} {flags}" + if label is None: + label = f"{args}" + else: + logging.getLogger(__name__).critical(f"\n=======\nSUBPROC {args}\n=======") + + reset_progress_step(description=f"{label}", prefix="[bold green]") + + # args = shlex.split(args) + + env = os.environ.copy() + pythonpath = env.pop("PYTHONPATH", None) + + if single_thread: + env["MKL_NUM_THREADS"] = "1" + env["OMP_NUM_THREADS"] = "1" + env["OPENBLAS_NUM_THREADS"] = "1" + env["NUMBA_NUM_THREADS"] = "1" + env["VECLIB_MAXIMUM_THREADS"] = "1" + env["NUMEXPR_NUM_THREADS"] = "1" + + if multi_thread: + env["MKL_NUM_THREADS"] = str(multi_thread.get("MKL", 1)) + env["OMP_NUM_THREADS"] = str(multi_thread.get("OMP", 1)) + env["OPENBLAS_NUM_THREADS"] = str(multi_thread.get("OPENBLAS", 1)) + env["NUMBA_NUM_THREADS"] = str(multi_thread.get("NUMBA", 1)) + env["VECLIB_MAXIMUM_THREADS"] = str(multi_thread.get("VECLIB", 1)) + env["NUMEXPR_NUM_THREADS"] = str(multi_thread.get("NUMEXPR", 1)) + + # if pythonpath: + # print(f"removed PYTHONPATH from ENV: {pythonpath}") + # else: + # print(f"no removed PYTHONPATH from ENV!") + # + # for k, v in env.items(): + # print(f" - {k}: {v}") + + # if conda_prefix is not None: + # args = ["conda", "init", "bash", "&&", 'conda', 'activate', conda_prefix, '&&'] + list(args) + # args = ['conda', 'run', '-p', conda_prefix] + list(args) + + if conda_prefix: + conda_prefix_1 = os.environ.get("CONDA_PREFIX_1", None) + if conda_prefix_1 is None: + conda_prefix_1 = os.environ.get("CONDA_PREFIX", None) + if os.name == "nt": + conda_prefix_1 = conda_prefix_1.replace("\\", "/") + conda_prefix = conda_prefix.replace("\\", "/") + joiner = "; " + else: + joiner = " && " + script = [ + f"source {conda_prefix_1}/etc/profile.d/conda.sh", + f'conda activate "{conda_prefix}"', + args, + ] + args = "bash -c '" + joiner.join(script) + "'" + + process = _stream_subprocess( + args=args, + cwd=cwd, + env=env, + ) + + # don't swallow any error, because it's the Step swallow decorator + # responsibility to decide to ignore or not. + if process.returncode: + raise subprocess.CalledProcessError( + process.returncode, + process.args, + process.stdout, + process.stderr, + ) diff --git a/activitysim/workflows/steps/title.py b/activitysim/workflows/steps/title.py new file mode 100644 index 0000000000..8dc7baf600 --- /dev/null +++ b/activitysim/workflows/steps/title.py @@ -0,0 +1,10 @@ +from .progression import update_progress_overall +from .wrapping import workstep + + +@workstep +def title( + label="ActivitySim Workflow", + formatting="bold blue", +) -> None: + update_progress_overall(label, formatting) diff --git a/activitysim/workflows/steps/update_yaml.py b/activitysim/workflows/steps/update_yaml.py new file mode 100644 index 0000000000..079eb878f7 --- /dev/null +++ b/activitysim/workflows/steps/update_yaml.py @@ -0,0 +1,50 @@ +from pypyr.errors import KeyNotInContextError +from pypyr.steps.fetchyaml import run_step as _fetch +from pypyr.steps.filewriteyaml import run_step as _write +from pypyr.steps.py import run_step as _run_step + +from .progression import progress, progress_overall, progress_step + + +def run_step(context): + """Update yaml file with payload. + + If you do not set encoding, will use the system default, which is utf-8 + for everything except windows. + + Args: + context: pypyr.context.Context. Mandatory. + The following context keys expected: + - updateYaml + - path. mandatory. path-like. Read and write file + here. + - payload. Add this to output file. + - encoding. string. Defaults None (platform default, + usually 'utf-8'). + + Returns: + None. + + Raises: + pypyr.errors.KeyNotInContextError: fileWriteYaml or + fileWriteYaml['path'] missing in context. + pypyr.errors.KeyInContextHasNoValueError: fileWriteYaml or + fileWriteYaml['path'] exists but is None. + + """ + context.assert_key_has_value("updateYaml", __name__) + fetch_yaml_input = context.get_formatted("updateYaml") + context["fetchYaml"] = { + "path": fetch_yaml_input["path"], + "key": "file_payload", + "encoding": fetch_yaml_input.get("encoding", None), + } + _fetch(context) + payload = fetch_yaml_input["payload"] + context["file_payload"].update(payload) + context["fileWriteYaml"] = { + "path": fetch_yaml_input["path"], + "payload": context["file_payload"], + "encoding": fetch_yaml_input.get("encoding", None), + } + _write(context) diff --git a/activitysim/workflows/steps/wrapping.py b/activitysim/workflows/steps/wrapping.py new file mode 100644 index 0000000000..ca3f48c8ba --- /dev/null +++ b/activitysim/workflows/steps/wrapping.py @@ -0,0 +1,204 @@ +import importlib +import logging +from inspect import getfullargspec +from typing import Mapping + +from pypyr.context import Context + +from . import get_formatted_or_default +from .error_handler import error_logging +from .progression import reset_progress_step + +logger = logging.getLogger(__name__) + + +class MockReport: + def __enter__(self): + pass + + def __exit__(self, exc_type, exc_val, exc_tb): + if exc_type or exc_val or exc_tb: + logger.exception(repr(exc_val)) + + def __lshift__(self, other): + logger.info(str(other)) + + +class workstep: + """ + Decorator for functions that write report sections or update context. + + The decorator will generate a `run_step` function in the same module, + wrapped with additional arguments and appropriately annotated for use + with the pypyr workflow model. The original function also remains + available to import and use without changes. + + When called as a step inside a pypyr workflow, the following context + variables are potentially accessed: + + report : xmle.Reporter + The active report into which new figures or tables are added. + caption : str + A caption for the item being processed. This is used both in + writing out the output (if any) in the report and for logging + step progression during a run. + caption_type : str + The caption type (typically, 'fig' for figures or 'tab' + for tables). + progress_tag : str + Use this instead of `caption` to log step progression during a run. + + If the function returns values that should update the context, that + can be done in one of three ways: + + - Set `updates_context` to True and return a `dict`, and use that + dict to update the context directly. + - Return a single object, and set `returns_names` to a string + giving the name that object should take in the context. + - Return a sequence of objects, and set `returns_names` to a + matching sequence of names that those objects should take + in the context. + + Otherwise, the return value is appended to the report. To declare that + there is no return value and no reporting should be done, you must + explicitly annotate the function with a return value of `-> None`. + + Important: there can be only one `workstep` in + each module. If you need more than one, make another separate module. + + Parameters + ---------- + wrapped_func : Callable + returns_names : str or tuple[str], optional + updates_context : bool, default False + + Returns + ------- + wrapped_func : Callable + The original wrapped function + + """ + + def __new__(cls, wrapped_func=None, *, returns_names=None, updates_context=False): + """ + Initialize a work step wrapper. + + Parameters + ---------- + wrapped_func : Callable + The function being decorated. + """ + if isinstance(wrapped_func, (str, tuple, list)): + # the returns_names are provided instead of the wrapped func + returns_names = wrapped_func + wrapped_func = None + self = super().__new__(cls) + self._returns_names = returns_names + self._updates_context = updates_context + if wrapped_func is not None: + return self(wrapped_func) + else: + return self + + def __call__(self, wrapped_func): + returns_names = self._returns_names + updates_context = self._updates_context + ( + _args, + _varargs, + _varkw, + _defaults, + _kwonlyargs, + _kwonlydefaults, + _annotations, + ) = getfullargspec(wrapped_func) + + if isinstance(returns_names, str): + returns_names = (returns_names,) + + def run_step(context: Context = None) -> None: + + caption = get_formatted_or_default(context, "caption", None) + progress_tag = get_formatted_or_default(context, "progress_tag", caption) + if progress_tag is not None: + reset_progress_step(description=progress_tag) + + return_type = _annotations.get("return", "") + _updates_context = updates_context or return_type in {dict, Context} + if return_type not in {None, dict, Context}: + if returns_names is None and not _updates_context: + context.assert_key_has_value( + key="report", caller=wrapped_func.__module__ + ) + report = context.get("report", MockReport()) + with report: + caption_type = get_formatted_or_default(context, "caption_type", "fig") + caption_maker = get_formatted_or_default(context, caption_type, None) + # parse and run function itself + if _defaults is None: + ndefault = 0 + _required_args = _args + else: + ndefault = len(_defaults) + _required_args = _args[:-ndefault] + args = [] + for arg in _required_args: + context.assert_key_has_value( + key=arg, caller=wrapped_func.__module__ + ) + try: + args.append(context.get_formatted_or_raw(arg)) + except Exception as err: + raise ValueError(f"extracting {arg} from context") from err + if ndefault: + for arg, default in zip(_args[-ndefault:], _defaults): + args.append(get_formatted_or_default(context, arg, default)) + kwargs = {} + for karg in _kwonlyargs: + if karg in _kwonlydefaults: + kwargs[karg] = get_formatted_or_default( + context, karg, _kwonlydefaults[karg] + ) + else: + context.assert_key_has_value( + key=karg, caller=wrapped_func.__module__ + ) + try: + kwargs[karg] = context.get_formatted_or_raw(karg) + except Exception as err: + raise ValueError(f"extracting {karg} from context") from err + if _varkw: + kwargs.update(context) + for arg in _required_args: + if arg in kwargs: + kwargs.pop(arg) + outcome = error_logging(wrapped_func)(*args, **kwargs) + if returns_names: + if len(returns_names) == 1: + context[returns_names[0]] = outcome + else: + for returns_name, out in zip(returns_names, outcome): + context[returns_name] = out + elif updates_context: + if not isinstance(outcome, Mapping): + raise ValueError( + f"{wrapped_func.__name__} is marked as updates_context, " + f"it should return a mapping" + ) + context.update(outcome) + elif outcome is not None: + caption_level = get_formatted_or_default( + context, "caption_level", None + ) + if caption is not None: + report << caption_maker(caption, level=caption_level) + report << outcome + + module = importlib.import_module(wrapped_func.__module__) + if hasattr(module, "run_step"): + raise ValueError( + f"{wrapped_func.__module__}.run_step exists, there can be only one per module" + ) + setattr(module, "run_step", run_step) + + return wrapped_func diff --git a/conda-environments/activitysim-dev-base.yml b/conda-environments/activitysim-dev-base.yml new file mode 100644 index 0000000000..166503e34b --- /dev/null +++ b/conda-environments/activitysim-dev-base.yml @@ -0,0 +1,72 @@ +# Environment for Advanced Development +# This is a set of recommended dependencies for ActivitySim developers. +# It includes a variety of tools and packages not necessary for actually +# running models, but useful for writing code, running tests and +# experiments, etc. +# +# This file does not install ActivitySim or Sharrow, you must do that yourself +# (so that you can install a preferred branch or fork) +# +# usage: $ mamba env create --file=activitysim-dev-base.yml -n ASIM-DEV + +channels: +- conda-forge +- nodefaults +dependencies: +- python=3.9 +- pip +- asv # for benchmarking +- black +- bump2version # for making a release +- coveralls +- cytoolz >= 0.8.1 +- dask +- descartes +- filelock +- fsspec +- geopandas +- gh +- git +- ipykernel # so this env will appear in jupyter as a selection +- isort +- jupyterlab +- larch >= 5.5.8 +- matplotlib +- myst-parser # allows markdown in sphinx +- nbconvert +- nbformat +- numba >= 0.55.2 +- numexpr +- numpy >= 1.16.1,<=1.21 +- numpydoc +- openmatrix >= 0.3.4.1 +- orca >= 1.6 +- pandas >= 1.1.0 +- pre-commit +- psutil >= 4.1 +- pyarrow >= 2.0 +- pycodestyle +- pydantic +- pydata-sphinx-theme +- pyinstrument +- pypyr +- pytables >=3.5.1,<3.7 +- pytest +- pytest-cov +- pytest-regressions +- pyyaml >= 5.1 +- requests >= 2.7 +- rich +- ruby # required for benchmarking pre-commit hooks +- setuptools_scm +- simwrapper > 1.7 +- snakeviz # for profiling +- sphinx +- sphinx_rtd_theme +- sphinx-argparse +- xarray >= 0.21 +- xmle +- zarr + +- pip: + - autodoc_pydantic diff --git a/conda-environments/activitysim-dev.yml b/conda-environments/activitysim-dev.yml index 373e03c555..ba52658896 100644 --- a/conda-environments/activitysim-dev.yml +++ b/conda-environments/activitysim-dev.yml @@ -1,11 +1,18 @@ # Environment for development -# usage: $ conda env create --file=activitysim-dev.yml -n ASIM-DEV-TMP +# This is a set of recommended dependencies for ActivitySim developers. +# It includes a variety of tools and packages not necessary for actually +# running models, but useful for writing code, running tests and +# experiments, etc. +# +# usage: $ mamba env create --file=activitysim-dev.yml -n ASIM-SH channels: - conda-forge +- nodefaults dependencies: - python=3.9 - pip - asv # for benchmarking +- black - bump2version # for making a release - coveralls - cytoolz >= 0.8.1 @@ -17,13 +24,14 @@ dependencies: - gh - git - ipykernel # so this env will appear in jupyter as a selection +- isort - jupyterlab - larch >= 5.5.8 - matplotlib - myst-parser # allows markdown in sphinx - nbconvert - nbformat -- numba >= 0.51.2 +- numba >= 0.55.2 - numexpr - numpy >= 1.16.1,<=1.21 - numpydoc @@ -34,6 +42,7 @@ dependencies: - psutil >= 4.1 - pyarrow >= 2.0 - pycodestyle +- pydantic - pydata-sphinx-theme - pyinstrument - pypyr @@ -45,6 +54,8 @@ dependencies: - requests >= 2.7 - rich - ruby # required for benchmarking pre-commit hooks +- setuptools_scm +- sharrow >= 2.3.2 - simwrapper > 1.7 - snakeviz # for profiling - sphinx @@ -55,4 +66,5 @@ dependencies: - zarr - pip: + - autodoc_pydantic - -e .. diff --git a/conda-environments/activitysim-test-larch.yml b/conda-environments/activitysim-test-larch.yml deleted file mode 100644 index 5f4b1f6871..0000000000 --- a/conda-environments/activitysim-test-larch.yml +++ /dev/null @@ -1,32 +0,0 @@ -# Environment for testing estimation in travis -# usage: $ conda env create --file=activitysim-test.yml -name: asimtest -channels: -- conda-forge -dependencies: -- python=${TRAVIS_PYTHON_VERSION} -- pip -- black -- coveralls -- cytoolz >= 0.8.1 -- isort -- larch >=5.5.3 -- nbmake -- numba >= 0.55.2 -- numpy >= 1.16.1,<=1.21 -- openmatrix >= 0.3.4.1 -- orca >= 1.6 -- pandas >= 1.1.0 -- psutil >= 4.1 -- pyarrow >= 2.0 -- pypyr >= 5.3 -- pytables >= 3.5.1,<3.7 # orca's constraint -- pytest -- pytest-cov -- pytest-regressions -- pyyaml >= 5.1 -- requests >= 2.7 -- sharrow >= 2.2.4 -- simwrapper > 1.7 -- xarray >= 0.21 -- zarr diff --git a/conda-environments/activitysim-test.yml b/conda-environments/activitysim-test.yml deleted file mode 100644 index bcd8e7d400..0000000000 --- a/conda-environments/activitysim-test.yml +++ /dev/null @@ -1,30 +0,0 @@ -# Environment for testing in travis -# usage: $ conda env create --file=activitysim-test.yml -name: asimtest -channels: -- conda-forge -dependencies: -- python=${TRAVIS_PYTHON_VERSION} -- black -- coveralls -- cytoolz >= 0.8.1 -- isort -- nbmake -- numba >= 0.55.2 -- numpy >= 1.16.1,<=1.21 -- openmatrix >= 0.3.4.1 -- orca >= 1.6 -- pandas >= 1.1.0 -- psutil >= 4.1 -- pyarrow >= 2.0 -- pypyr >= 5.3 -- pytables >= 3.5.1,<3.7 # orca's constraint -- pytest -- pytest-cov -- pytest-regressions -- pyyaml >= 5.1 -- requests >= 2.7 -- sharrow >= 2.2 -- simwrapper > 1.7 -- xarray >= 0.21 -- zarr diff --git a/conda-environments/docbuild.yml b/conda-environments/docbuild.yml index a4617fcf60..d1c526725c 100644 --- a/conda-environments/docbuild.yml +++ b/conda-environments/docbuild.yml @@ -1,4 +1,5 @@ # Environment for building docs +# This environment contains the set of dependencies needed to build the documentation. name: docbuild channels: - conda-forge @@ -18,7 +19,7 @@ dependencies: - larch >=5.5.3 - matplotlib - myst-parser -- numba >= 0.51.2 +- numba >= 0.55.2 - numpy >= 1.16.1,<=1.21 - numpydoc - openmatrix >= 0.3.4.1 @@ -34,7 +35,7 @@ dependencies: - pytest-regressions - pyyaml >= 5.1 - requests >= 2.7 -- sharrow >= 2.2.4 +- sharrow >= 2.3.2 - simwrapper > 1.7 - sphinx-argparse - sphinx_rtd_theme diff --git a/conda-environments/github-actions-tests.yml b/conda-environments/github-actions-tests.yml index 01108bdc48..399236fc97 100644 --- a/conda-environments/github-actions-tests.yml +++ b/conda-environments/github-actions-tests.yml @@ -1,4 +1,7 @@ # Environment for testing in GitHub Actions +# This environment contains a minimal set of dependencies needed to run most tests. +# It does not install ActivitySim itself (which is done by the test scripts) and +# is not meant for use outside the CI tools. name: asim-test channels: - conda-forge @@ -23,7 +26,7 @@ dependencies: - pytest-regressions - pyyaml >= 5.1 - requests >= 2.7 -- sharrow >= 2.2.4 +- sharrow >= 2.3.2 - simwrapper > 1.7 - xarray >= 0.21 - zarr diff --git a/docs/dev-guide/index.rst b/docs/dev-guide/index.rst index e18240afad..5a1e8ad92d 100644 --- a/docs/dev-guide/index.rst +++ b/docs/dev-guide/index.rst @@ -23,6 +23,8 @@ Contents :maxdepth: 3 install + using-sharrow + workflows ../development ../models ../core diff --git a/docs/dev-guide/using-sharrow.md b/docs/dev-guide/using-sharrow.md new file mode 100644 index 0000000000..2b0f56aafe --- /dev/null +++ b/docs/dev-guide/using-sharrow.md @@ -0,0 +1,87 @@ + +# Using Sharrow + +This page will walk through an exercise of running a model with `sharrow`. + +## How it Works + +Sharrow accelerates ActivitySim in part by using numba to create optimized and +pre-compiled versions of utility specification files, and caching those bits +of code to disk. + +```{important} +Running the compiler needs to be done in single-process mode, otherwise the +various process all do the compiling and compete to write to the same cache +location on disk, which is likely to fail. You can safely run in +multiprocessing mode after all the compilation for all model components is +complete. +``` + + +## Getting the Code + +We'll assume that activitysim and sharrow have been installed in editable +mode, per the [developer install instructions](install.md). + +The code to implement `sharrow` in `activitysim` is in the +[sharrow-black](https://github.com/camsys/activitysim/tree/sharrow-black) branch +of the [camsys/activitysim](https://github.com/camsys/activitysim) repository. +If you have checked out from this repository directly, you can just switch to +the correct branch: + +```sh +cd activitysim +git switch sharrow-black +cd .. +``` + +If your local Git repository is configured with only the main ActivitySim +parent repo as its remote, you can find the code in +[PR #579](https://github.com/ActivitySim/activitysim/pull/579), accessible on the +command line using the `gh` tool: + +```sh +cd activitysim +gh pr checkout 579 # use `gh auth login` first if needed +cd .. +``` + + +## Prototype MTC Examples + +Testing with sharrow requires two steps: test mode and production mode. + +In test mode, the code is run to compile all the spec files and +ascertain whether the functions are working correctly. Production mode +can then just run the pre-compiled functions with sharrow, which is much +faster. + +You can run both, plus the ActivitySim in "legacy" mode that does not use sharrow, +as well as a reference implementation (version 1.0.4), all together using the +"mini" dataset for testing in one workflow. + +```sh +activitysim workflow sharrow-contrast/mtc_mini +``` + +Alternatively, you can use the full size skims. To test this model with +100k households and full skims (1475 zones), you can run the "mtc_full" workflow: + +```sh +activitysim workflow sharrow-contrast/mtc_full +``` + +To use the full synthetic population as well, run the multiprocess workflow: + +```sh +activitysim workflow sharrow-contrast/mtc_mp +``` + +Lastly, a comprehensive performance testing suite on the prototype MTC model +(warning: this takes hours!) + +```sh +activitysim workflow sharrow-contrast/mtc_comprehensive +``` + +All these performance tests assume you have sufficient RAM to run without chunking. diff --git a/docs/dev-guide/workflows.md b/docs/dev-guide/workflows.md new file mode 100644 index 0000000000..5cacc42ba8 --- /dev/null +++ b/docs/dev-guide/workflows.md @@ -0,0 +1,21 @@ + +# Workflows + +ActivitySim workflows use the [`pypyr`](https://pypyr.io/) library. This +task runner is more flexible than orca, and relies on isolated 'context' +namespaces rather than storing tables and configurations in Python's +global state. Workflows are activated using the `activitysim workflow` +command line interface. + +You can run a workflow from a local working directory, or one of +a number of pre-packaged workflows that are included with ActivitySim. + +A collection of workflows used to compare the new *sharrow* code against +legacy implementations can be found [here](https://github.com/camsys/activitysim/tree/sharrow-black/activitysim/workflows/sharrow-contrast). + +To invoke a pre-packaged workflow, you can provide the workflow's +name and location within the `activitysim/activitysim/workflows` directory. + +```shell +activitysim workflow sharrow-contrast/mtc_mini +``` diff --git a/docs/development.rst b/docs/development.rst index fef5684c31..1c891115ba 100644 --- a/docs/development.rst +++ b/docs/development.rst @@ -8,44 +8,44 @@ Software Design --------------- The core software components of ActivitySim are described below. ActivitySim is -implemented in Python, and makes heavy use of the vectorized backend C/C++ libraries in -`pandas `__ and `numpy `__ in order to be quite performant. -The core design principle of the system is vectorization of for loops, and this principle -is woven into the system wherever reasonable. As a result, the Python portions of the software -can be thought of as more of an orchestrator, data processor, etc. that integrates a series of -C/C++ vectorized data table and matrix operations. The model system formulates -each simulation as a series of vectorized table operations and the Python layer +implemented in Python, and makes heavy use of the vectorized backend C/C++ libraries in +`pandas `__ and `numpy `__ in order to be quite performant. +The core design principle of the system is vectorization of for loops, and this principle +is woven into the system wherever reasonable. As a result, the Python portions of the software +can be thought of as more of an orchestrator, data processor, etc. that integrates a series of +C/C++ vectorized data table and matrix operations. The model system formulates +each simulation as a series of vectorized table operations and the Python layer is responsible for setting up and providing expressions to operate on these large data tables. -In developing this software platform, we strive to adhere to a best practices approach to scientific computing, +In developing this software platform, we strive to adhere to a best practices approach to scientific computing, as summarized in `this article. `__ Model Orchestrator ~~~~~~~~~~~~~~~~~~ An ActivitySim model is a sequence of model / data processing steps, commonly known as a data pipeline. -A well defined data pipeline has the ability to resume jobs at a known point, which facilitates +A well defined data pipeline has the ability to resume jobs at a known point, which facilitates debugging of problems with data and/or calculations. It also allows for checkpointing model resources, such as the state of each person at a point in the model simulation. Checkpointing also allows for regression testing of results at specified points in overall model run. -ActivitySim's model orchestrator makes use of depedency injection, which is where one object (or method) +ActivitySim's model orchestrator makes use of depedency injection, which is where one object (or method) supplies the dependencies of another object. Dependency injection is done by the :mod:`activitysim.core.inject` -module, which wraps `ORCA `__, an orchestration/pipeline tool. Inject defines model -steps, dynamic data sources, and connects them to processing functions. It also defines dynamic data tables -based on pandas DataFrames, columns based on pandas Series, and injectables (functions). Model steps are executed +module, which wraps `ORCA `__, an orchestration/pipeline tool. Inject defines model +steps, dynamic data sources, and connects them to processing functions. It also defines dynamic data tables +based on pandas DataFrames, columns based on pandas Series, and injectables (functions). Model steps are executed as steps registered with the model orchestration engine. Over time Inject has extended ORCA's functionality by -adding a :ref:`pipeline_in_detail` that runs a series of model steps, manages the state of the data -tables throughout the model run, allows for restarting at any model step, and integrates with the +adding a :ref:`pipeline_in_detail` that runs a series of model steps, manages the state of the data +tables throughout the model run, allows for restarting at any model step, and integrates with the random number generation procedures (see :ref:`random_in_detail`). Data Handling ~~~~~~~~~~~~~ -ActivitySim works with three open data formats, `HDF5 `__ -, `Open Matrix (OMX) `__, and `CSV `__ . +ActivitySim works with three open data formats, `HDF5 `__ +, `Open Matrix (OMX) `__, and `CSV `__ . The HDF5 binary data container is used for the :ref:`pipeline_in_detail` data store. -OMX, which is based on HDF5, is used for input and output matrices (skims and demand matrices). CSV files +OMX, which is based on HDF5, is used for input and output matrices (skims and demand matrices). CSV files are used for various inputs and outputs as well. Three key data structures in ActivitySim are: @@ -58,34 +58,34 @@ Expressions ~~~~~~~~~~~ ActivitySim exposes all model expressions in CSV files. These model expression CSV files -contain Python expressions, mainly pandas/numpy expression, and reference input data tables -and network skim matrices. With this design, the Python code, which can be thought of as a generic expression -engine, and the specific model calculations, such as the utilities, are separate. This helps to avoid -modifying the actual Python code when making changes to the models, such as during model calibration. An -example of model expressions is found in the example auto ownership model specification file - +contain Python expressions, mainly pandas/numpy expression, and reference input data tables +and network skim matrices. With this design, the Python code, which can be thought of as a generic expression +engine, and the specific model calculations, such as the utilities, are separate. This helps to avoid +modifying the actual Python code when making changes to the models, such as during model calibration. An +example of model expressions is found in the example auto ownership model specification file - `auto_ownership.csv `__. Refer to the :ref:`expressions` section for more detail. -Many of the models have pre- and post-processor table annotators, which read a CSV file of expression, calculate -required additional table fields, and join the fields to the target tables. An example table annotation expressions -file is found in the example configuration files for households for the CDAP model - +Many of the models have pre- and post-processor table annotators, which read a CSV file of expression, calculate +required additional table fields, and join the fields to the target tables. An example table annotation expressions +file is found in the example configuration files for households for the CDAP model - `annotate_households_cdap.csv `__. Refer to :ref:`table_annotation` for more information and the :func:`activitysim.abm.models.util.expressions.assign_columns` function. Choice Models ~~~~~~~~~~~~~ -ActivitySim currently supports multinomial (MNL) and nested logit (NL) choice models. Refer to :ref:`logit_in_detail` -for more information. It also supports custom expressions as noted above, which can often be used to -code additional types of choice models. In addition, developers can write their own choice models -in Python and expose these through the framework. +ActivitySim currently supports multinomial (MNL) and nested logit (NL) choice models. Refer to :ref:`logit_in_detail` +for more information. It also supports custom expressions as noted above, which can often be used to +code additional types of choice models. In addition, developers can write their own choice models +in Python and expose these through the framework. Person Time Windows ~~~~~~~~~~~~~~~~~~~ -The departure time and duration models require person time windows. Time windows are adjacent time -periods that are available for travel. ActivitySim maintains time windows in a pandas table where each row is -a person and each time period is a column. As travel is scheduled throughout the simulation, the relevant +The departure time and duration models require person time windows. Time windows are adjacent time +periods that are available for travel. ActivitySim maintains time windows in a pandas table where each row is +a person and each time period is a column. As travel is scheduled throughout the simulation, the relevant columns for the tour, trip, etc. are updated as needed. Refer to :ref:`time_windows` for more information. Models @@ -103,7 +103,7 @@ typically does the following: * runs a data postprocessor on the output table data that needs additional fields for later models * writes the resulting table data to the pipeline -See :ref:`models` for more information. +See :ref:`models` for more information. Development Install @@ -129,14 +129,14 @@ ActivitySim development adheres to the following standards. Style ~~~~~ -* Python code should follow the `pycodestyle style guide `__ +* Python code should follow the `black code style `__ * Python docstrings should follow the `numpydoc documentation format `__ Imports ~~~~~~~ * Imports should be one per line. -* Imports should be grouped into standard library, third-party, and intra-library imports. +* Imports should be grouped into standard library, third-party, and intra-library imports. * ``from`` import should follow regular ``imports``. * Within each group the imports should be alphabetized. * Imports of scientific Python libraries should follow these conventions: @@ -150,7 +150,7 @@ Imports Working Together in the Repository ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -We use `GitHub Flow `__. The key points to +We use `GitHub Flow `__. The key points to our GitHub workflow are: * The ``main`` branch contains the latest release version of the ActivitySim resources @@ -197,9 +197,11 @@ ActivitySim uses the following `versioning convention `__, a tool to check Python code against the pycodestyle style conventions +* `black `__, a tool to check and enforce black + code style on Python code +* `isort ` to organize imports * `pytest `__, a Python testing tool * `coveralls `__, a tool for measuring code coverage and publishing code coverage stats online @@ -207,18 +209,18 @@ To run the tests locally, first make sure the required packages are installed. :: - pycodestyle + black --check py.test These same tests are run by Travis with each push to the repository. These tests need to pass in order to merge the revisions into main. -In some cases, test targets need to be updated to match the new results produced by the code since these -are now the correct results. In order to update the test targets, first determine which tests are -failing and then review the failing lines in the source files. These are easy to identify since each -test ultimately comes down to one of Python's various types of `assert` statements. Once you identify -which `assert` is failing, you can work your way back through the code that creates the test targets in -order to update it. After updating the test targets, re-run the tests to confirm the new code passes all +In some cases, test targets need to be updated to match the new results produced by the code since these +are now the correct results. In order to update the test targets, first determine which tests are +failing and then review the failing lines in the source files. These are easy to identify since each +test ultimately comes down to one of Python's various types of `assert` statements. Once you identify +which `assert` is failing, you can work your way back through the code that creates the test targets in +order to update it. After updating the test targets, re-run the tests to confirm the new code passes all the tests. See :ref:`adding_agency_examples` for more information on testing, most notably, agency example models. @@ -226,28 +228,28 @@ See :ref:`adding_agency_examples` for more information on testing, most notably, Profiling ~~~~~~~~~ -A handy way to profile ActivitySim model runs is to use `snakeviz `__. -To do so, first install snakeviz and then run ActivitySim with the Python profiler (cProfile) to create +A handy way to profile ActivitySim model runs is to use `snakeviz `__. +To do so, first install snakeviz and then run ActivitySim with the Python profiler (cProfile) to create a profiler file. Then run snakeviz on the profiler file to interactively explore the component runtimes. Documentation ~~~~~~~~~~~~~ -The documentation is written in `reStructuredText `__ markup +The documentation is written in `reStructuredText `__ markup and built with `Sphinx `__. In addition to converting rst files to html and other document formats, these tools also read the inline Python docstrings and convert them into html as well. ActivitySim's docstrings are written in `numpydoc format -`__ since it is easier to use +`__ since it is easier to use than standard rst format. -To build the documentation, first make sure the required packages are installed. Next, build the +To build the documentation, first make sure the required packages are installed. Next, build the documentation in html format with the ``make html`` command run from the ``docs`` folder. -If the activitysim package is installed, then the documentation will be built from that version of -the source code instead of the git repo version. When pushing revisions to the repo, the documentation +If the activitysim package is installed, then the documentation will be built from that version of +the source code instead of the git repo version. When pushing revisions to the repo, the documentation is automatically built by Travis after successfully passing the tests. -GitHub automatically publishes the gh-pages branch at https://activitysim.github.io/activitysim. +GitHub automatically publishes the gh-pages branch at https://activitysim.github.io/activitysim. .. _release_steps : @@ -262,26 +264,26 @@ document. Issues and Support ~~~~~~~~~~~~~~~~~~ -Issue tracking and support is done through GitHub `issues `__. +Issue tracking and support is done through GitHub `issues `__. License ~~~~~~~ -ActivitySim is provided "as is." See the +ActivitySim is provided "as is." See the `License `__ for more information. Contribution Review Criteria ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -When contributing to ActivitySim, the set of questions below will be asked of the contribution. Make sure to also -review the documentation above before making a submittal. The automated test system also provides some helpful +When contributing to ActivitySim, the set of questions below will be asked of the contribution. Make sure to also +review the documentation above before making a submittal. The automated test system also provides some helpful information where identified. -To submit a contribution for review, issue a pull request with a comment introducing your contribution. The comment -should include a brief overview, responses to the questions, and pointers to related information. The entire submittal -should ideally be self contained so any additional documentation should be in the pull request as well. -The `PMC `__ and/or its Contractor will handle the review request, comment on each -question, complete the feedback form, and reply to the pull request. If accepted, the commit(s) will +To submit a contribution for review, issue a pull request with a comment introducing your contribution. The comment +should include a brief overview, responses to the questions, and pointers to related information. The entire submittal +should ideally be self contained so any additional documentation should be in the pull request as well. +The `PMC `__ and/or its Contractor will handle the review request, comment on each +question, complete the feedback form, and reply to the pull request. If accepted, the commit(s) will be `squashed and merged `__. Its a good idea to setup a pre-submittal meeting to discuss questions and better understand expectations. @@ -292,10 +294,10 @@ Its a good idea to setup a pre-submittal meeting to discuss questions and better 3. Are the runtimes reasonable and does it provide documentation justifying this claim? 4. Does it include non-Python code, such as C/C++? If so, does it compile on any OS and are compilation instructions included? 5. Is it licensed with the ActivitySim license that allows the code to be freely distributed and modified and includes attribution so that the provenance of the code can be tracked? Does it include an official release of ownership from the funding agency if applicable? - 6. Does it appropriately interact with the data pipeline (i.e. it doesn't create new ways of managing data)? + 6. Does it appropriately interact with the data pipeline (i.e. it doesn't create new ways of managing data)? 7. Does it include regression tests to enable checking that consistent results will be returned when updates are made to the framework? - 8. Does it include sufficient test coverage and test data for existing and proposed features? - 9. Any other comments or suggestions for improving the developer experience? + 8. Does it include sufficient test coverage and test data for existing and proposed features? + 9. Any other comments or suggestions for improving the developer experience? **Feedback** @@ -303,7 +305,7 @@ The PMC and/or its Contractor will provide feedback for each review criteria abo +-----------------------------------+-------------+-------------------+-------------------+ | Status | Code | Documentation | Tests/Examples | -+===================================+=============+===================+===================+ ++===================================+=============+===================+===================+ | Accept | | | | +-----------------------------------+-------------+-------------------+-------------------+ | Accept but recommend revisions | | | | @@ -403,7 +405,7 @@ to maintain the dictionary of the examples and how to get and run them. Running the Test System ~~~~~~~~~~~~~~~~~~~~~~~ -The automatic TravisCI test system runs the test examples and the cropped agency examples. Examples of the testing +The automatic TravisCI test system runs the test examples and the cropped agency examples. Examples of the testing resources for each agency example that need to be up-to-date are: * `scripts folder (including crop script) `_ @@ -438,7 +440,7 @@ To help understand this case, the inclusion of :ref:`placeholder_psrc` as an age When an agency example includes new submodels and/or contributions to the core that need to be reviewed and then pulled/accepted: * First, the agency example must comply with the steps outlined above under "When an agency wants to update their example". -* Second, the agency example must be up-to-date with the latest develop version of the code so the revisions to the code are only the exact revisions for the new submodels and/or contributions to the core. +* Second, the agency example must be up-to-date with the latest develop version of the code so the revisions to the code are only the exact revisions for the new submodels and/or contributions to the core. * The new submodels and/or contributions to the core will then be reviewed by the repository manager and it's likely some revisions will be required for acceptance. Key items in the review include python code, user documentation, and testable examples for all new components. If the contribution is just new submodels, then the agency example that exercises the new submodel is sufficient for test coverage since TravisCI will automatically test the cropped version of the new submodel. If the contribution includes revisions to the core that impact other test examples, then the developer is responsible for ensuring all the other tests that are up-to-date are updated/passing as well. This includes other agency examples that are up-to-date. This is required to ensure the contribution to the core is adequately complete. To help understand this case, the addition of the parking location choice model for :ref:`prototype_arc` is discussed. First, ARC gets their example in good working order - i.e. updates to develop, makes any required revisions to their model to get it working, creates a cropped and full scaled example, and creates the expected test results. In addition, this use case includes additional submodel and/or core code so ARC also authors the new feature, including documentation and any other relevant requirements such as logging, tracing, support for estimation, etc. With the new example and feature working offline, then ARC issues a pull request to add prototype_arc and the new submodel/core code and makes sure the automatic tests are passing. Once accepted, the automatic test system will run the test example tests and the cropped agency examples. Since the new feature - parking location choice model - is included in prototype_arc, then new feature is now tested. Any testing of downstream impacts from the parking location choice model would also need to be implemented in the example. diff --git a/docs/examples.rst b/docs/examples.rst index cf2431efc2..d9ebb72f54 100644 --- a/docs/examples.rst +++ b/docs/examples.rst @@ -507,13 +507,13 @@ Area types The prototype_mtc_sf inputs were created by the ``other_resources\scripts\create_sf_example.py`` script, which creates the land use, synthetic population, and skim inputs for a subset of user-defined zones. -.. index:: configuration Configuration ^^^^^^^^^^^^^ This section has been moved to :ref:`configuration`. + .. _sub-model-spec-files: Sub-Model Specification Files @@ -1487,8 +1487,8 @@ prototype_semcog The prototype_semcog added a :ref:`work_from_home`, :ref:`telecommute_frequency`, :ref:`transit_pass_subsidy` and :ref:`transit_pass_ownership` submodel. These submodel specification files are below, and are in addition to the :ref:`prototype_mtc` submodel :ref:`sub-model-spec-files`. These submodels were added to prototype_semcog as extensions, which is a way for users to add -submodels within their model setup as opposed to formally adding them to the activitysim package. Extension submodels are run through -the `models` settings. However, the model must be run with the `simulation.py` script instead of the command line interface +submodels within their model setup as opposed to formally adding them to the activitysim package. Extension submodels are run through +the `models` settings. However, the model must be run with the `simulation.py` script instead of the command line interface in order to load the extensions folder. .. _semcog-sub-model-spec-files: diff --git a/docs/gettingstarted.rst b/docs/gettingstarted.rst index 00db3e792b..8f75c21e62 100644 --- a/docs/gettingstarted.rst +++ b/docs/gettingstarted.rst @@ -65,7 +65,7 @@ Additional libraries can also be installed later. You may want to consider thes tools for certain development tasks:: # packages for testing - mamba install pytest pytest-cov coveralls pycodestyle pytest-regressions -c conda-forge --override-channels -n asim + mamba install pytest pytest-cov coveralls black flake8 pytest-regressions -c conda-forge --override-channels -n asim # packages for building documentation mamba install sphinx numpydoc sphinx_rtd_theme==0.5.2 -c conda-forge --override-channels -n asim @@ -172,11 +172,11 @@ To setup and run the primary example (see :ref:`examples`), do the following: * Review the outputs in the output directory .. note:: - Common configuration settings can be overridden at runtime. See ``activitysim -h``, ``activitysim create -h`` and ``activitysim run -h``. + Common configuration settings can be overridden at runtime. See ``activitysim -h``, ``activitysim create -h`` and ``activitysim run -h``. ActivitySim model runs can be configured with settings file inheritance to avoid duplicating settings across model configurations. See :ref:`cli` for more information. Additional examples, including the full scale prototype MTC regional demand model, estimation integration examples, multiple zone system examples, -and examples for agency partners are available for creation by typing ``activitysim create -l``. To create these examples, ActivitySim downloads the (large) input files from +and examples for agency partners are available for creation by typing ``activitysim create -l``. To create these examples, ActivitySim downloads the (large) input files from the `ActivitySim resources `__ repository. See :ref:`examples` for more information. Try the Notebooks @@ -213,7 +213,7 @@ on the amount of RAM and number of processors allocated. See :ref:`multiprocess .. note:: ActivitySim has been run in the cloud, on both Windows and Linux using - `Microsoft Azure `__. Example configurations, + `Microsoft Azure `__. Example configurations, scripts, and runtimes are in the ``other_resources\example_azure`` folder. .. _mkl_settings : diff --git a/docs/models.rst b/docs/models.rst index 00dd28aa4a..9ea71ab708 100644 --- a/docs/models.rst +++ b/docs/models.rst @@ -118,12 +118,12 @@ the coefficient adjustment at each iteration is: ``new_coefficient = log( target_percent / current_percent ) + current_coefficient``. The main interface to the work from home model is the -:py:func:`~activitysim.examples.prototype_semcog.extensions.work_from_home` function. This +:py:func:`~activitysim.abm.models.work_from_home` function. This function is registered as an Inject step in the example Pipeline. Core Table: ``persons`` | Result Field: ``work_from_home`` | Skims Keys: NA -.. automodule:: activitysim.examples.prototype_semcog.extensions.work_from_home +.. automodule:: activitysim.abm.models.work_from_home :members: .. _school_location: @@ -248,12 +248,12 @@ person :ref:`transit_pass_ownership` model and the tour and trip mode choice mod via fare discount adjustments. The main interface to the transit pass subsidy model is the -:py:func:`~activitysim.examples.prototype_semcog.extensions.transit_pass_subsidy` function. This +:py:func:`~activitysim.abm.models.transit_pass_subsidy` function. This function is registered as an Inject step in the example Pipeline. Core Table: ``persons`` | Result Field: ``transit_pass_subsidy`` | Skims Keys: NA -.. automodule:: activitysim.examples.prototype_semcog.extensions.transit_pass_subsidy +.. automodule:: activitysim.abm.models.transit_pass_subsidy :members: .. _transit_pass_ownership: @@ -268,12 +268,12 @@ result of this model can be used to condition downstream models such as the tour mode choice models via fare discount adjustments. The main interface to the transit pass ownership model is the -:py:func:`~activitysim.examples.prototype_semcog.extensions.transit_pass_ownership` function. This +:py:func:`~activitysim.abm.models.transit_pass_ownership` function. This function is registered as an Inject step in the example Pipeline. Core Table: ``persons`` | Result Field: ``transit_pass_ownership`` | Skims Keys: NA -.. automodule:: activitysim.examples.prototype_semcog.extensions.transit_pass_ownership +.. automodule:: activitysim.abm.models.transit_pass_ownership :members: .. _auto_ownership: @@ -375,12 +375,12 @@ level of telecommuting. The model alternatives are the frequency of telecommutin days per week (0 days, 1 day, 2 to 3 days, 4+ days). The main interface to the work from home model is the -:py:func:`~activitysim.examples.prototype_semcog.extensions.telecommute_frequency` function. This +:py:func:`~activitysim.abm.models.telecommute_frequency` function. This function is registered as an Inject step in the example Pipeline. Core Table: ``persons`` | Result Field: ``telecommute_frequency`` | Skims Keys: NA -.. automodule:: activitysim.examples.prototype_semcog.extensions.telecommute_frequency +.. automodule:: activitysim.abm.models.telecommute_frequency :members: .. _freeparking: diff --git a/docs/users-guide/cli-tools.rst b/docs/users-guide/cli-tools.rst new file mode 100644 index 0000000000..84feb40704 --- /dev/null +++ b/docs/users-guide/cli-tools.rst @@ -0,0 +1,31 @@ +================== +Command Line Tools +================== + + +activitysim run +--------------- + +.. argparse:: + :module: activitysim.cli.main + :func: parser + :prog: activitysim + :path: run + + +activitysim create +------------------ + +.. argparse:: + :module: activitysim.cli.main + :func: parser + :prog: activitysim + :path: create + + -d --destination : @replace + Path to new project directory. If this directory already exists, the + newly created example will be copied to a subdirectory within the + existing directory, and named according to the example name. Otherwise, + a new directory is created with this name and the newly created example + will be copied directly into it. + diff --git a/docs/users-guide/configuration.rst b/docs/users-guide/configuration.rst index e726301e64..2da0e31e47 100644 --- a/docs/users-guide/configuration.rst +++ b/docs/users-guide/configuration.rst @@ -19,6 +19,10 @@ component. Settings InputTable + OutputTable + OutputTables + MultiprocessStep + MultiprocessStepSlice NetworkSettings TAZ_Settings ZarrDigitalEncoding diff --git a/docs/users-guide/index.rst b/docs/users-guide/index.rst index 266d01aa45..0bffb2f9f3 100644 --- a/docs/users-guide/index.rst +++ b/docs/users-guide/index.rst @@ -24,6 +24,7 @@ Contents :maxdepth: 3 ../gettingstarted + cli-tools ../examples configuration ../cli diff --git a/other_resources/scripts/verify_results.py b/other_resources/scripts/verify_results.py index 9c8fb8e326..d5868ae165 100644 --- a/other_resources/scripts/verify_results.py +++ b/other_resources/scripts/verify_results.py @@ -88,7 +88,7 @@ ############################################################# # read distance matrix (DIST) -distmat = omx.open_file(distance_matrix_filename)["DIST"][:] +distmat = omx.open_file(distance_matrix_filename, mode="r")["DIST"][:] ############################################################# # EXPORT TABLES diff --git a/setup.cfg b/setup.cfg index bf02916f30..4b878a9e87 100644 --- a/setup.cfg +++ b/setup.cfg @@ -35,7 +35,7 @@ install_requires = pypyr >= 5.3 pyyaml >= 5.1 requests >= 2.7 - sharrow >= 2.2.4 + sharrow >= 2.3.2 simwrapper > 1.7 tables >= 3.5.1 xarray >= 0.21