RecycleItems task not running frequently enough

[Gobberwart]

EDIT: I have investigated this, and have made modifications to the code to address the issue. Currently seeking assistance with submitting changes for review if anyone can assist?

Expected Behavior

Bot should recycle items every few minutes, keeping no more than the specified number of each item type.

Actual Behavior

ItemRecycler appears very rarely in the log so bag fills up with pokeballs unnecessarily.

Config.json has the following config for RecycleItems:

        "type": "RecycleItems",
        "config": {
          "min_empty_space": 15,
          "max_balls_keep": 150,
          "max_potions_keep": 50,
          "max_berries_keep": 70,
          "max_revives_keep": 70,
          "item_filter": {
            "Pokeball":       { "keep" : 30 },
            "Greatball":      { "keep" : 50 },
            "Ultraball":      { "keep" : 50 },
            "Potion":         { "keep" : 10 },
            "Super Potion":   { "keep" : 20 },
            "Hyper Potion":   { "keep" : 20 },
            "Max Potion":     { "keep" : 20 },
            "Revive":         { "keep" : 20 },
            "Razz Berry":     { "keep" : 50 }
          },
          "recycle_wait_min": 1,
          "recycle_wait_max": 4

Output shows:

2016-08-21 06:20:02,674 [ItemRecycler] [INFO] [item_discarded] Discarded 35x Pokeball.
2016-08-21 06:20:06,369 [ItemRecycler] [INFO] [item_discarded] Discarded 30x Potion.
2016-08-21 06:20:08,690 [ItemRecycler] [INFO] [item_discarded] Discarded 17x Super Potion.
2016-08-21 06:20:12,459 [ItemRecycler] [INFO] [item_discarded] Discarded 95x Pokeball.
2016-08-21 06:20:16,467 [ItemRecycler] [INFO] [item_discarded] Discarded 2x Super Potion.
2016-08-21 06:20:19,161 [ItemRecycler] [INFO] [item_discarded] Discarded 8x Hyper Potion.
2016-08-21 06:20:21,640 [ItemRecycler] [INFO] [item_discarded] Discarded 16x Revive.

Next update of items shows:

2016-08-21 06:21:36,885 [UpdateLiveInventory] [INFO] [show_inventory] Items: 147/350 | Pokeballs: 35 | GreatBalls: 24 | UltraBalls: 2 | RazzBerries: 4 | LuckyEgg: 4

This appears to have worked, but it's been over an hour and the ItemRecycler task has not run since, so now I have:

2016-08-21 07:23:00,780 [UpdateLiveInventory] [INFO] [show_inventory] Items: 324/350 | Pokeballs: 143 | GreatBalls: 20 | UltraBalls: 2 | RazzBerries: 2 | LuckyEgg: 4

Update: The ItemRecycler task just happened again, but it was more than an hour between runs.

Your FULL config.json

{
    "auth_service": "ptc",
    "username": "REDACTED",
    "password": "REDACTED",
    "gmapkey": "REDACTED",
    "location": "-37.814574, 144.962164",
    "libencrypt_location": "",
    "logging_color": true,
    "tasks": [
      {
        "type": "HandleSoftBan"
      },
      {
        "type": "SleepSchedule",
        "config": {
          "enabled": false,
          "time": "22:54",
          "duration":"7:46",
          "time_random_offset": "00:24",
          "duration_random_offset": "00:43"
        }
      },
      {
        "type": "RandomPause",
        "config": {
          "enabled": false,
          "min_duration": "00:00:10",
          "max_duration": "00:10:00",
          "min_interval": "00:10:00",
          "max_interval": "02:00:00"
        }
      },
      {
        "type": "CompleteTutorial",
        "config": {
          "enabled": false,
          "// set a name": "",
          "nickname": ""
        }
      },
      {
        "type": "CollectLevelUpReward"
      },
      {
        "type": "IncubateEggs",
        "config": {
          "longer_eggs_first": true
        }
      },
      {
        "type": "UpdateLiveStats",
        "config": {
          "enabled": true,
          "min_interval": 10,
          "stats": ["username", "uptime", "stardust_earned", "xp_earned", "xp_per_hour", "stops_visited"],
          "terminal_log": true,
          "terminal_title": true
        }
      },
      {
        "type": "UpdateLiveInventory",
        "config": {
          "enabled": true,
          "min_interval": 120,
          "show_all_multiple_lines": false,
          "items": ["space_info", "pokeballs", "greatballs", "ultraballs", "razzberries", "luckyegg"]
        }
      },
      {
        "type": "TransferPokemon"
      },
      {
        "type": "NicknamePokemon",
        "config": {
          "enabled": false,
          "nickname_template": "{iv_pct}_{iv_ads}"
        }
      },
      {
        "type": "EvolvePokemon",
        "config": {
            "evolve_all": "all",
            "first_evolve_by": "cp",
            "evolve_above_cp": 700,
            "evolve_above_iv": 0.8,
            "logic": "or",
            "evolve_speed": 20,
            "use_lucky_egg": false
        }
      },
      {
        "type": "RecycleItems",
        "config": {
          "min_empty_space": 15,
          "max_balls_keep": 150,
          "max_potions_keep": 50,
          "max_berries_keep": 70,
          "max_revives_keep": 70,
          "item_filter": {
            "Pokeball":       { "keep" : 30 },
            "Greatball":      { "keep" : 50 },
            "Ultraball":      { "keep" : 50 },
            "Potion":         { "keep" : 10 },
            "Super Potion":   { "keep" : 20 },
            "Hyper Potion":   { "keep" : 20 },
            "Max Potion":     { "keep" : 20 },
            "Revive":         { "keep" : 20 },
            "Razz Berry":     { "keep" : 50 }
          },
          "recycle_wait_min": 1,
          "recycle_wait_max": 4
        }
      },
      {
        "type": "CatchPokemon",
        "config": {
          "catch_visible_pokemon": true,
          "catch_lured_pokemon": true,
          "min_ultraball_to_keep": 5,
          "catch_throw_parameters": {
            "excellent_rate": 0.1,
            "great_rate": 0.5,
            "nice_rate": 0.3,
            "normal_rate": 0.1,
            "spin_success_rate" : 0.6
          },
          "catch_simulation": {
            "flee_count": 3,
            "flee_duration": 2,
            "catch_wait_min": 2,
            "catch_wait_max": 6,
            "berry_wait_min": 2,
            "berry_wait_max": 3,
            "changeball_wait_min": 2,
            "changeball_wait_max": 3
          }
        }
      },
      {
        "type": "SpinFort",
        "config": {
          "spin_wait_min": 2,
          "spin_wait_max": 3
        }
      },
      {
        "type": "MoveToFort",
        "config": {
          "lure_attraction": true,
          "lure_max_distance": 2000
        }
      }
    ],
    "map_object_cache_time": 5,
    "forts": {
      "avoid_circles": true,
      "max_circle_size": 50
    },
    "websocket_server": false,
    "walk_max": 6.24,
    "walk_min": 2.16,
    "debug": false,
    "test": false,
    "health_record": true,
    "location_cache": false,
    "distance_unit": "km",
    "reconnecting_timeout": 15,
    "daily_catch_limit": 800,
    "action_wait_min": 1,
    "action_wait_max": 4,
    "catch_randomize_reticle_factor": 1.0,
    "catch_randomize_spin_factor": 1.0,
    "catch": {
      "any": {"catch_above_cp": 0, "catch_above_iv": 0, "logic": "or"},
      "// Example of always catching Rattata:": {},
      "// Rattata": { "always_catch" : true }
    },
    "release": {
      "any": {"keep_best_cp": 2},
      "// Example of always releasing Rattata:": {},
      "// Rattata": {"always_release": true},
      "// Example of keeping 3 stronger (based on CP) Pidgey:": {},
      "// Pidgey": {"keep_best_cp": 3},
      "// Example of keeping 2 stronger (based on IV) Zubat:": {},
      "// Zubat": {"keep_best_iv": 2},
      "// Also, it is working with any": {},
      "// any": {"keep_best_iv": 3},
      "// Example of keeping the 2 strongest (based on CP) and 3 best (based on IV) Zubat:": {},
      "// Zubat": {"keep_best_cp": 2, "keep_best_iv": 3}
    },
    "vips" : {
         "Any pokemon put here directly force to use Berry & Best Ball to capture, to secure the capture rate!": {},
        "any": {"catch_above_cp": 1200, "catch_above_iv": 0.9, "logic": "or" },
        "Lapras": {},
        "Moltres": {},
        "Zapdos": {},
        "Articuno": {},

        "// S-Tier pokemons (if pokemon can be evolved into tier, list the representative)": {},
        "Mewtwo": {},
        "Dragonite": {},
        "Snorlax": {},
        "// Mew evolves to Mewtwo": {},
        "Mew": {},
        "Arcanine": {},
        "Vaporeon": {},
        "Gyarados": {},
        "Exeggutor": {},
        "Muk": {},
        "Weezing": {},
        "Flareon": {}

    }
}

Output when issue occurred

N/A (See above)

Steps to Reproduce

Run the bot and wait.

Other Information

OS: Windows 10
Branch: dev
Git Commit: cf4b458
Python Version: 2.7

[Gobberwart]

Just had a look through the code (recycle_items.py) and it appears as though the RecycleItems task will only run if space in the bag is less than min_empty_space:

    def should_run(self):
        """
        Returns a value indicating whether the recycling process should be run.
        :return: True if the recycling process should be run; otherwise, False.
        :rtype: bool
        """
        if inventory.Items.get_space_left() <= (DEFAULT_MIN_EMPTY_SPACE if self.min_empty_space is None else self.min_empty_space):
            return True
        return False

At least I understand why this happens, but it's causing my bag to fill up with basic pokeballs to the detriment of everything else.

At this stage, I've just added a random chance of forcing the recycle every 20 bot cycles (give or take), but that's unbelievably inelegant.

Can this be set to run on a timer regardless of whether min_empty_space has been reached?

[Gobberwart]

I've added a feature to my code which allows scheduling of a "forced" recycle at a random interval between a given min and mix duration (set in config.json), regardless of whether min_empty_space is reached. Tested and working well.

There are three changed files, recycle_items.py, colored_logging_handler.py and init.py, plus additions to config examples.

Can someone please advise what I should do to submit my code for review/inclusion in the official dev branch? (sorry, this is the first time I've worked on a community git project).

[Gregwar]

👍
Looks like this also causes the web/inventory-*.json not being refreshed

[Gobberwart]

I believe Gregwar is correct. As it stands, the only time web inventory is updated is when RecycleItems is triggered, which is currently only when your bag is almost full.

Even then, it only updates after the categories are processed, but not after going through the individual items in the filters, which means it may still be incorrect if you have individual "keep" settings for various items.

Looking further into this, I have a couple of issues with the way the item recycling code currently works as follows:

1. The recycler should run more often (as above).
2. Individual items should be processed before categories.
3. Inventory should refresh more frequently and accurately, either at the very end of each recycle (when items are actually recycled), or on every bot cycle (regardless of whether or not items are recycled - not sure what impact this might have on 'human-like' behaviour though)
4. The recycler worker will currently only return the result of whatever the final task in recycling is, not an overall success/fail for all tasks.

I have a working copy of the code that addresses all these issues. Still need advice on how to submit for review.

[Gobberwart]

Re item 3 above, it looks like #3948 has something to do with this.

Added "inventory.refresh_inventory()" to the "tick" function in init.py (line 535) forces an inventory refresh every bot tick.

[Gobberwart]

Yes @mjmadsen, that is occurring, however the code has recently (a week or so ago) been changed so that recycling will only occur when space remaining reaches a given threshold. In my case, this is defined as "15" but the default is 6.

The bot should ideally recycle whenever any of the items in the filter list are above the count specified.

Additionally, the bot's inventory is only updated when recycling occurs, leading to major discrepancies between the bot's internal inventory and the web interface.

As mentioned, I have made modifications to my code to address this. Would appreciate advice on how to get this code reviewed as I don't have permission to create a branch, create a pull request etc.

[BriceSD]

Take a look at the doc, it explains how to recycle every tick.
Regarding the rest, I don't have access to a computer, so I can't look further. But I'm pretty sure this task doesn't write in a file.

[Gobberwart]

I don't know exactly what you're referring to, but the doc(s) included with the distribution does not explain how to recycle every tick. Even if some doc somewhere explains this, the code does not currently support running every tick at all.

The "work" function of recycle_item.py calls the "should_run" function, and the only criteria for running is whether or not min_empty_space is exceeded.

        if inventory.Items.get_space_left() <= (DEFAULT_MIN_EMPTY_SPACE if self.min_empty_space is None else self.min_empty_space):
            return True
        return False

I suppose you could force it to run every tick by setting min_empty_space to something like 250, but that would be a bit silly.

I'm also absolutely sure this task doesn't write to a file. It does, however, call inventory.refresh, which writes its output to a file, specifically web/inventory-%username%.json.

Nothing else in the code currently calls inventory.refresh, hence the huge delays in updating the web interface.

Still looking for advice regarding submitting code for review, thanks.

Also very happy to be proven wrong :)

[mjmadsen]

@Gobberwart You can "fork" this by clicking it in the top right corner. Then you can work in your own space with the current commit. When you finish making changes to the dev branch you can go to "Pull Request" to submit your changes to our dev branch.

[BriceSD]

Thank you for pointing out this inventory refresh. It shouldn’t be there anymore, looks like I forgot one when resolving conflicts.

We’re moving away from writing in files. With our new map we won’t need it anymore. That being said, it’s not ready yet. I guess there is quite a lot of people who want this. Refreshing the inventory at every tick could be a temporary solution.

About the documentation you can find it on our homepage.

You should change your config if you want RecycleItems to run more frequently. Increase the min_empty_space or decrease amounts to keep in inventory. Or both. That’s why we have a customizable config file after all.

[Gobberwart]

OK, I've submitted a pull request #4513. Would appreciate any feedback.