Collapsed forwarding support in ATS core

[ezelkow1]

This adds a new write fail action, CACHE_WL_FAIL_ACTION_COLLAPSED_FORWARDING. If it fails to obtain a write lock then it will loop for open_write_retries*open_read_retry_time, each time doing a DecideCacheLookup. In the end if it fails to ever either obtain a
write lock or find the object to read then it does the default write fail action of going to the origin.

In testing using duplicated requests I can see multiple simultaneous requests end up in this read loop and eventually get a hit in cache while the original request gets written. Occaisonally there is still an extra hit or two to the origin which I believe is
state machine dependant, so sort of unavoidable, sometimes you can have the original request finish the write while a duplicated request ends up obtaining the write lock directly afterwards and so it hits the origin as well. This may be unavoidable unless cache
lookups are done everywhere before a write.

On an origin fail it will do the default of sending back an ERR_CONNECT_FAIL

[sudheerv]

case CACHE_EVENT_OPEN_WRITE_FAILED:
// if (master_sm->t_state.txn_conf->cache_open_write_fail_action == //CACHE_OPEN_WRITE_FAIL_ACTION::READ_RETRY) {
// just fall back to open_read_retry settings
//open_write_tries = master_sm->t_state.txn_conf->max_cache_open_write_retries;
//}

if (open_write_tries <= master_sm->t_state.txn_conf->max_cache_open_write_retries) {
  // Retry open write;
  open_write_cb = false;
  do_schedule_in();
} else {

[sudheerv]

change this to
// the idea is to go back to read state on write fail (as opposed to relying on write_fail action)
// master_sm->handleEvent(CACHE_EVENT_OPEN_READ, data)

// make sure CACHE_EVENT_OPEN_READ in state_cache_open_write is handled without crashing (looks for stale object, in this case we may not have one)
HttpSM::state_cache_open_write(int event, void *data)

case CACHE_EVENT_OPEN_READ:
// The write vector was locked and the cache_sm retried
// and got the read vector again.
if (t_state.cache_info.object_read) {
cache_sm.cache_read_vc->get_http_info(&t_state.cache_info.object_read);
// ToDo: Should support other levels of cache hits here, but the cache does not support it (yet)
if (cache_sm.cache_read_vc->is_ram_cache_hit()) {
t_state.cache_info.hit_miss_code = SQUID_HIT_RAM;
} else {
t_state.cache_info.hit_miss_code = SQUID_HIT_DISK;
}

ink_assert(t_state.cache_info.object_read != nullptr);
t_state.source = HttpTransact::SOURCE_CACHE;
**}**
// clear up CACHE_LOOKUP_MISS, let Freshness function decide
// hit status
t_state.cache_lookup_result         = HttpTransact::CACHE_LOOKUP_NONE;
t_state.cache_info.write_lock_state = HttpTransact::CACHE_WL_READ_RETRY;
break;

[ezelkow1]

open_write_fail.txt

Should try this patch, missed some state updates in the cache_open_read event for write_lock

[bryancall]

[approve ci]

[ezelkow1]

superseded by #6028 since @sudheerv had a chance to work on it more