MARS, meet Python

Jarred Green - MPP

20 Feb 2024

Outline

A [very] brief intro to MAGIC analysis
What is magicpy?
Why magicpy?
Features and future

MAGIC Analysis

Easy mode: no moonlight

Download data
Train a random forest (RF) to pick out gammas from background
Apply RF to data and do science

MAGIC Analysis

Easy mode: no moonlight

In reality this is not easy and usually not fun
ROOT/C++ code only via command line and text files

MAGIC Analysis

Hard mode: moon analysis.

with multiple moon levels.

Just keeping track of your directory structure is hard enough here!

MAGIC Analysis

Answer: magicpy

[one day soon]

What is magicpy?

An easy and fun way to interact with MAGIC analysis!

MARS programs wrapped in python!

import magicpy.analysis as analysis
import magicpy.periods as periods

source = analysis.Source(
    name="CrabNebula",  
    period=periods.ST0316
)

source.search()


"""
Total runs: 124, Total hours: 31.55

Period: ST0316, Total runs: 118, Total hours: 30.37
+----------+------------+------------+------------+------------+
| ST0316   |   za35to50 |   za05to35 |   za50to62 |   za62to70 |
|----------+------------+------------+------------+------------|
| weak     |       4.17 |      11.46 |       5.27 |       4.26 |
| moderate |       1.21 |       0.24 |       0    |       0    |
| decent   |       1.12 |       0.97 |       0.09 |       0    |
| strong   |       0.03 |       0    |       0    |       0    |
| high     |       0    |       1.55 |       0    |       0    |
+----------+------------+------------+------------+------------+
"""

What is magicpy?

An easy and fun way to interact with MAGIC analysis!

MARS programs wrapped in python!

import magicpy as mp

source = mp.analysis.Source(
    name="CrabNebula",  
    period=mp.periods.ST0316
)

source.search()


"""
Total runs: 124, Total hours: 31.55

Period: ST0316, Total runs: 118, Total hours: 30.37
+----------+------------+------------+------------+------------+
| ST0316   |   za35to50 |   za05to35 |   za50to62 |   za62to70 |
|----------+------------+------------+------------+------------|
| weak     |       4.17 |      11.46 |       5.27 |       4.26 |
| moderate |       1.21 |       0.24 |       0    |       0    |
| decent   |       1.12 |       0.97 |       0.09 |       0    |
| strong   |       0.03 |       0    |       0    |       0    |
| high     |       0    |       1.55 |       0    |       0    |
+----------+------------+------------+------------+------------+
"""

Viewing runs/subruns with pandas

source.runs

What is magicpy?

Can use magicpy commands in scripts

or from jupyter notebooks

import magicpy.analysis as analysis
import magicpy.periods as periods

source = analysis.Source(
    name="CrabNebula",  
    period=periods.ST0316
)

source.search()


"""
Total runs: 124, Total hours: 31.55

Period: ST0316, Total runs: 118, Total hours: 30.37
+----------+------------+------------+------------+------------+
| ST0316   |   za35to50 |   za05to35 |   za50to62 |   za62to70 |
|----------+------------+------------+------------+------------|
| weak     |       4.17 |      11.46 |       5.27 |       4.26 |
| moderate |       1.21 |       0.24 |       0    |       0    |
| decent   |       1.12 |       0.97 |       0.09 |       0    |
| strong   |       0.03 |       0    |       0    |       0    |
| high     |       0    |       1.55 |       0    |       0    |
+----------+------------+------------+------------+------------+
"""

What is magicpy?

"""A few random features"""
# download data

source.get_data(
  datalevel=mp.datalevels.Calibrated
)

# get runbooks
source.download_runbooks()

# run analysis steps
source.run_star()

# or just run MARS programs with ease

superstar = mp.mars.SuperStar(
  files="/path/to/files/*.root"
)

superstar.set_parameter(
  "MJSuperStar.RuncardThing", "yas"
)

superstar.run()

Why magicpy?

Fully manual analysis

Tons of directories and variables to keep track of
Can take months to understand all the tiny details
You might go crazy

Why magicpy?

Fully manual analysis

autoMAGIC

Amazing auto pipeline
SOBO still wants a human cross check
Needs infrastructure

Tons of directories and variables to keep track of
Can take months to understand all the tiny details
You might go crazy

Why magicpy?

Fully manual analysis

Tons of directories and variables to keep track of
Can take months to understand all the tiny details
You might go crazy

Every PhD / Master student spends way to much time coding their own pipeline

Instead of doing science!

Amazing auto pipeline
SOBO still wants a human cross check
Needs infrastructure

autoMAGIC

Why magicpy?

Fully manual analysis

Tons of directories and variables to keep track of
Can take months to understand all the tiny details
You might go crazy

⚠️ lots of manual effort
⚠️ ROOT/C++
✅ very customizable
✅ can run anywhere

Amazing auto pipeline
SOBO still wants a human cross check
Needs infrastructure

autoMAGIC

✅ super automatic!
✅ lots of python
✅ MARS as backend
⚠️ need infrastructure

Why magicpy?

Fully manual analysis

Tons of directories and variables to keep track of
Can take months to understand all the tiny details
You might go crazy

⚠️ lots of manual effort
⚠️ ROOT/C++
✅ very customizable
✅ can run anywhere

✨magicpy

✅ a bit of both worlds
✅ less wasted time for FAs
✅ snake!
✅ can run on your laptop

✅ super automatic!
✅ lots of python
✅ MARS as backend
⚠️ need infrastructure

Amazing auto pipeline
SOBO still wants a human cross check
Needs infrastructure

autoMAGIC

Minimal example - moon + dark analysis

A fast analysis

Minimal example - moon + dark analysis

yes, that's it

Performing fast analysis for CrabNebula

Current data distribution 
Total runs: 11, Total hours: 2.41
Period: ST0316, Total runs: 11, Total hours: 2.41
+----------+------------+
| ST0316   |   za35to50 |
|----------+------------|
| weak     |       0.96 |
| decent   |       0.79 |
| strong   |       0.66 |
+----------+------------+


Downloading data at level SuperStar to /remote/magicdata5/jgreen/MAGICPY_ANALYSIS/ALL/SuperStar
Downloading 11 files from 3 directories.
100%|██████████| 11/11 [00:00<00:00, 44837.07it/s]
Downloading data at level Calibrated to /remote/magicdata5/jgreen/MAGICPY_ANALYSIS/ALL/Calibrated

Downloading 222 files from 6 directories.
100%|██████████| 222/222 [00:00<00:00, 54116.91it/s]

Checking cleaning and moon levels for CrabNebula...
100%|██████████| 107/107 [00:00<00:00, 10562.26it/s]

Data distribution after checking moon levels
Total runs: 11, Total hours: 2.41
Period: ST0316, Total runs: 11, Total hours: 2.41
+----------+------------+
| ST0316   |   za35to50 |
|----------+------------|
| weak     |       0.96 |
| decent   |       0.79 |
| strong   |       0.66 |
+----------+------------+

Running star on 49 runs...
Running star on CrabNebula in parallel with 12 cores.
100%|██████████| 98/98 [00:04<00:00, 21.61it/s]

Running superstar on 49 runs...
Running superstar on CrabNebula in parallel with 12 cores.
100%|██████████| 7/7 [00:00<00:00, 32.11it/s]

Random Forest for ST0316 za35to50 weak moon found: /jgreen/MAGICPY_ANALYSIS/RFs/
Random Forest for ST0316 za35to50 strong moon found: /jgreen/MAGICPY_ANALYSIS/RFs/
Random Forest for ST0316 za35to50 decent moon found: /jgreen/MAGICPY_ANALYSIS/RFs/
All RFs found. Running melibea for CrabNebula.

Making theta2 plots...
Making skymaps...
Done analyzing CrabNebula ✨

source.go() logs   --->

(it works)

No direct contact with MARS/command line

Other features

automatic fast analysis
Dark and moon analysis
Easily search database
Download and organize data
Filter data by any parameter
Read and scan runbooks
Easily create RFs
Search and filter OFF data
Download MC data
Automatically avoid redownloading or reprocessing data

Easily apply moon cleanings
Run MARS programs in parallel
Run single MARS executables from python
Edit/change runcards/CLI params from within python
Lightcurves, spectra, and theta2
Save analyses and resume later
Add noise to dark OFF data for moon analysis

already implemented

In the works for v1.0

Add parallelization everywhere possible
Finish implementing PIC mode
- doesn't download data
- uses condor to run jobs
Tool to browse/search for sources
Fine tune auto mode for fast analyses
Finish mono analysis implementation
Validate analyses
Add strict typing with mypy
Create good documentation
DL3
Interact with SOBO often

(just a selection shown here)

Release?

Still a couple more major features to implement
Still need pipeline validation

To answer most questions:

Can magicpy do *specific analysis thing*?

Does magicpy take into account *random analysis detail*?

Yes.

It's actively in development so we can ensure it does everything properly.

Let's discuss!

🧙🐍

A small demo

import magicpy.analysis as analysis
import magicpy.periods as periods

source = analysis.Source(
    name="CrabNebula",  
    period=periods.ST0316
)

source.search()


"""
Total runs: 124, Total hours: 31.55

Period: ST0316, Total runs: 118, Total hours: 30.37
+----------+------------+------------+------------+------------+
| ST0316   |   za35to50 |   za05to35 |   za50to62 |   za62to70 |
|----------+------------+------------+------------+------------|
| weak     |       4.17 |      11.46 |       5.27 |       4.26 |
| moderate |       1.21 |       0.24 |       0    |       0    |
| decent   |       1.12 |       0.97 |       0.09 |       0    |
| strong   |       0.03 |       0    |       0    |       0    |
| high     |       0    |       1.55 |       0    |       0    |
+----------+------------+------------+------------+------------+
"""

Starting an analysis

A small demo

Download and view runbooks

import datetime as dt

source.download_runbooks()


# get runbook text
source.get_runbook(
  date=dt.date(2021, 3, 16)
)


# scans runbook for your MOLA results
source.runbook_report()

"""
Scanning for CrabNebula:

------------------------------ opening runbook for 2021-02-03 ------------------------------

MOLA results:

t_eff: 6450.9 s
LE: 14.7 sigma
HE: 37.1 sigma
HE flux: 1.0 C.U.
----------------------------------- done with 2021-02-03 -----------------------------------


------------------------------ opening runbook for 2021-03-14 ------------------------------

Runbook does not follow standard setup, or source CrabNebula is somehow written differently.
----------------------------------- done with 2021-03-14 -----------------------------------
"""

A small demo

Filtering data with pandas / downloading

import magicpy.moons as moons
import magicpy.zeniths as zeniths

source.filter(
    {
        "za": [zeniths.za35to50],
        "moon_level": [moons.WeakMoon, moons.ModerateMoon, moons.DecentMoon],
        "L1_rate": ">0.0",
        "sum_trigger": "No_Sumt"
    }
)


# download data
source.get_data()


"""
Downloading data at level SuperStar to /remote/.../ALL/SuperStar
Downloading 11 files from 3 directories.
100%|██████████| 11/11 [00:00<00:00, 9067.87it/s]


Downloading data at level Calibrated to /remote/.../ALL/Calibrated
Downloading 222 files from 6 directories.
100%|██████████| 222/222 [00:00<00:00, 58137.83it/s]
"""

A small demo

moon data: apply proper cleaning levels

calculates rms moon levels
knows what cleaning levels to use per run
on a run-by-run bases

source.run_star()
source.run_superstar()

# 100%|██████████| 98/98 [00:25:38<00:00, 3.86s/it]

# or 

source.run_star(parallel=True)
source.run_superstar(parallel=True)

# apply random forest
source.apply_rfs()

"""
Random Forest for ST0316 za35to50 weak moon found: 
		/remote/.../RFs/ST0316/za35to50/weak

Running melibea for CrabNebula, ST0316, za35to50 and weak moon.
"""

A small demo

Train RFs in a few lines of code

import magicpy.analysis as analysis
import magicpy.periods as periods
import magicpy.moons as moons
import magicpy.zeniths as zeniths

PERIOD = periods.ST0316
ZENITH = zeniths.za35to50
MOON = moons.DecentMoon


off = analysis.Off(
    period=PERIOD, 
    zenith=ZENITH, 
    moon_level=MOON,
)


off.search()

# check combined MOLA results
# for OFF source ideas
off.get_mola()

A small demo

Train RFs in a few lines of code

Data selected :
	PKS2345-16            2 runs       0.6 h
	GB6-J0316+09          2 runs       0.5 h
	OriTrapezium          9 runs       2.9 h
	HB9SNR-R2             5 runs       1.7 h
	HydraA                2 runs       0.6 h
	GRB210321A            4 runs       1.3 h
	GRB210619             1 runs       0.1 h
	SGR1900+14            6 runs       1.8 h
	TXS0149+710           3 runs       0.5 h
	Total                34 runs      10.0 h

A small demo

Train RFs in a few lines of code

# create an MC instance as well

mc = analysis.MC(
  off_source=off
)

mc.get_data()
mc.run_star()
mc.run_superstar()


# train RFs
 
off.train(mc_data=mc)

# apply RFs to MCs

mc.apply_rfs()