WIP: explore MPS rewrite

[odow]

I started experimenting with #2420

It's a modest improvement:

(base) oscar@Oscars-MBP /tmp % julia --project=jmp ~/Downloads/generate-flow-oscar.jl --output_file mcflow.mps.gz --num_commodities 1000 --num_warehouses 20 --num_stores 200 --seed 1
Parsed args:
  output_file  =>  mcflow.mps.gz
  num_warehouses  =>  20
  num_factories_per_commodity  =>  5
  seed  =>  1
  additional_overtime_cost  =>  0.3
  num_stores  =>  200
  num_commodities  =>  1000
building model ...
model built
 33.586420 seconds (129.25 M allocations: 9.650 GiB, 19.94% gc time, 7.69% compilation time)

compared to #2418 (comment)

(base) oscar@Oscars-MBP /tmp % julia --project=opt ~/Downloads/generate-flow-oscar.jl --output_file mcflow.mps.gz --num_commodities 1000 --num_warehouses 20 --num_stores 200 --seed 1
Parsed args:
  output_file  =>  mcflow.mps.gz
  num_warehouses  =>  20
  num_factories_per_commodity  =>  5
  seed  =>  1
  additional_overtime_cost  =>  0.3
  num_stores  =>  200
  num_commodities  =>  1000
building model ...
model built
 41.327799 seconds (138.67 M allocations: 11.201 GiB, 15.64% gc time, 10.55% compilation time)

But it's a fairly large change, and I haven't added the quadratic, SOS, indicator support yet.

Another option is to use HiGHS instead:

import HiGHS
model = direct_model(HiGHS.Optimizer())
HiGHS.Highs_writeModel(JuMP.backend(model), parsed_args["output_file"])

resulting in

(base) oscar@Oscars-MBP /tmp % julia --project=jmp ~/Downloads/generate-flow-oscar.jl --output_file mcflow.mps.gz --num_commodities 1000 --num_warehouses 20 --num_stores 200 --seed 1
Parsed args:
  output_file  =>  mcflow.mps.gz
  num_warehouses  =>  20
  num_factories_per_commodity  =>  5
  seed  =>  1
  additional_overtime_cost  =>  0.3
  num_stores  =>  200
  num_commodities  =>  1000
building model ...
Running HiGHS 1.6.0: Copyright (c) 2023 HiGHS under MIT licence terms
model built
Writing the model to mcflow.mps.gz
WARNING: There are empty or excessively-long column names: using constructed names with prefix "c"
WARNING: There are empty or excessively-long row names: using constructed names with prefix "r"
 26.103356 seconds (51.45 M allocations: 4.525 GiB, 14.59% gc time, 4.10% compilation time)

The speed difference: 26 sec (HiGHS) compared to 33 sec (this PR) 41 sec (master). Half of the difference is rewriting the names for uniqueness. The other parts are odds and ends (Julia I/O writing to .gz?). So I think we're not far away from a C implementation.

For the memory: a bit misleading because @time doesn't measure the inner memory of HiGHS.

If needed to decrease memory usage further, you could also do something:

using JuMP, HiGHS
function build_model(N)
    model = direct_model(HiGHS.Optimizer())
    @variable(model, 0 <= x[i in 1:N] <= 1; integer = isodd(i))
    @constraint(model, [i in 2:N], x[i] + x[i-1] <= 1)
    @objective(model, Max, sum(i * x[i] for i in 1:N))
    return backend(model)
end

function main(N)
    model = build_model(N)
    GC.gc()    # Free all the JuMP stuff
    MOI.write_to_file(model, "file.mps")
    return
end
GC.gc(); @time main(100_000);

My conclusion is that we're probably okay for now, and that we can revisit this if it becomes a problem in future.

[odow]

I changed to constructing anonymous names R$i and C$i on demand, deleting the c_name::Vector{String} entirely. And. Things got worse? I don't really have an intuition the interaction of the GC and strings.

 34.762827 seconds (166.54 M allocations: 11.080 GiB, 16.20% gc time, 6.60% compilation time)

@mlubin, the other thing you could do to lower memory pressure (if you don't care about the long names) is:

set_string_names_on_creation(model, false)