MARS, meet Python

Jarred Green - MPP

20 Feb 2024

Outline

  • A [very] brief intro to MAGIC analysis
  • What is magicpy?
  • Why magicpy?
  • Features and future

MAGIC Analysis

Easy mode: no moonlight

  1. Download data
  2. Train a random forest (RF) to pick out gammas from background
  3. Apply RF to data and do science

MAGIC Analysis

Easy mode: no moonlight

  • In reality this is not easy and usually not fun
  • ROOT/C++ code only via command line and text files

MAGIC Analysis

Hard mode: moon analysis.

with multiple moon levels.

Just keeping track of your directory structure is hard enough here!

MAGIC Analysis

Answer: magicpy

[one day soon]

What is magicpy?

An easy and fun way to interact with MAGIC analysis!

MARS programs wrapped in python!

import magicpy.analysis as analysis
import magicpy.periods as periods

source = analysis.Source(
    name="CrabNebula",  
    period=periods.ST0316
)

source.search()


"""
Total runs: 124, Total hours: 31.55

Period: ST0316, Total runs: 118, Total hours: 30.37
+----------+------------+------------+------------+------------+
| ST0316   |   za35to50 |   za05to35 |   za50to62 |   za62to70 |
|----------+------------+------------+------------+------------|
| weak     |       4.17 |      11.46 |       5.27 |       4.26 |
| moderate |       1.21 |       0.24 |       0    |       0    |
| decent   |       1.12 |       0.97 |       0.09 |       0    |
| strong   |       0.03 |       0    |       0    |       0    |
| high     |       0    |       1.55 |       0    |       0    |
+----------+------------+------------+------------+------------+
"""

What is magicpy?

An easy and fun way to interact with MAGIC analysis!

MARS programs wrapped in python!

import magicpy as mp

source = mp.analysis.Source(
    name="CrabNebula",  
    period=mp.periods.ST0316
)

source.search()


"""
Total runs: 124, Total hours: 31.55

Period: ST0316, Total runs: 118, Total hours: 30.37
+----------+------------+------------+------------+------------+
| ST0316   |   za35to50 |   za05to35 |   za50to62 |   za62to70 |
|----------+------------+------------+------------+------------|
| weak     |       4.17 |      11.46 |       5.27 |       4.26 |
| moderate |       1.21 |       0.24 |       0    |       0    |
| decent   |       1.12 |       0.97 |       0.09 |       0    |
| strong   |       0.03 |       0    |       0    |       0    |
| high     |       0    |       1.55 |       0    |       0    |
+----------+------------+------------+------------+------------+
"""

Viewing runs/subruns with pandas

source.runs

What is magicpy?

What is magicpy?

Can use magicpy commands in scripts

or from jupyter notebooks

import magicpy.analysis as analysis
import magicpy.periods as periods

source = analysis.Source(
    name="CrabNebula",  
    period=periods.ST0316
)

source.search()


"""
Total runs: 124, Total hours: 31.55

Period: ST0316, Total runs: 118, Total hours: 30.37
+----------+------------+------------+------------+------------+
| ST0316   |   za35to50 |   za05to35 |   za50to62 |   za62to70 |
|----------+------------+------------+------------+------------|
| weak     |       4.17 |      11.46 |       5.27 |       4.26 |
| moderate |       1.21 |       0.24 |       0    |       0    |
| decent   |       1.12 |       0.97 |       0.09 |       0    |
| strong   |       0.03 |       0    |       0    |       0    |
| high     |       0    |       1.55 |       0    |       0    |
+----------+------------+------------+------------+------------+
"""

What is magicpy?

"""A few random features"""
# download data

source.get_data(
  datalevel=mp.datalevels.Calibrated
)

# get runbooks
source.download_runbooks()

# run analysis steps
source.run_star()

# or just run MARS programs with ease

superstar = mp.mars.SuperStar(
  files="/path/to/files/*.root"
)

superstar.set_parameter(
  "MJSuperStar.RuncardThing", "yas"
)

superstar.run()

Why magicpy?

Fully manual analysis

  • ​​​​​​Tons of directories and variables to keep track of
  • Can take months to understand all the tiny details
  • You might go crazy

Why magicpy?

Fully manual analysis

autoMAGIC

  • Amazing auto pipeline 
  • SOBO still wants a human cross check
  • Needs infrastructure 
  • ​​​​​​Tons of directories and variables to keep track of
  • Can take months to understand all the tiny details
  • You might go crazy

Why magicpy?

Fully manual analysis

  • ​​​​​​Tons of directories and variables to keep track of
  • Can take months to understand all the tiny details
  • You might go crazy

Every PhD / Master student spends way to much time coding their own pipeline

Instead of doing science!

  • Amazing auto pipeline 
  • SOBO still wants a human cross check
  • Needs infrastructure 

autoMAGIC

Why magicpy?

Fully manual analysis

  • ​​​​​​Tons of directories and variables to keep track of
  • Can take months to understand all the tiny details
  • You might go crazy
  • ⚠️ lots of manual effort
  • ⚠️ ROOT/C++
  • ✅ very customizable
  • ✅ can run anywhere
  • Amazing auto pipeline 
  • SOBO still wants a human cross check
  • Needs infrastructure 

autoMAGIC

  • ✅ super automatic!
  • ✅ lots of python
  • ✅ MARS as backend
  • ⚠️ need infrastructure

Why magicpy?

Fully manual analysis

  • ​​​​​​Tons of directories and variables to keep track of
  • Can take months to understand all the tiny details
  • You might go crazy
  • ⚠️ lots of manual effort
  • ⚠️ ROOT/C++
  • ✅ very customizable
  • ✅ can run anywhere

✨magicpy

  • ✅ a bit of both worlds
  • ✅ less wasted time for FAs
  • ✅ snake! 
  • ✅ can run on your laptop
  • ✅ super automatic!
  • ✅ lots of python
  • ✅ MARS as backend
  • ⚠️ need infrastructure
  • Amazing auto pipeline 
  • SOBO still wants a human cross check
  • Needs infrastructure 

autoMAGIC

Minimal example - moon + dark analysis

A fast analysis

A fast analysis

Minimal example - moon + dark analysis

yes, that's it

Performing fast analysis for CrabNebula

Current data distribution 
Total runs: 11, Total hours: 2.41
Period: ST0316, Total runs: 11, Total hours: 2.41
+----------+------------+
| ST0316   |   za35to50 |
|----------+------------|
| weak     |       0.96 |
| decent   |       0.79 |
| strong   |       0.66 |
+----------+------------+


Downloading data at level SuperStar to /remote/magicdata5/jgreen/MAGICPY_ANALYSIS/ALL/SuperStar
Downloading 11 files from 3 directories.
100%|██████████| 11/11 [00:00<00:00, 44837.07it/s]
Downloading data at level Calibrated to /remote/magicdata5/jgreen/MAGICPY_ANALYSIS/ALL/Calibrated

Downloading 222 files from 6 directories.
100%|██████████| 222/222 [00:00<00:00, 54116.91it/s]

Checking cleaning and moon levels for CrabNebula...
100%|██████████| 107/107 [00:00<00:00, 10562.26it/s]

Data distribution after checking moon levels
Total runs: 11, Total hours: 2.41
Period: ST0316, Total runs: 11, Total hours: 2.41
+----------+------------+
| ST0316   |   za35to50 |
|----------+------------|
| weak     |       0.96 |
| decent   |       0.79 |
| strong   |       0.66 |
+----------+------------+

Running star on 49 runs...
Running star on CrabNebula in parallel with 12 cores.
100%|██████████| 98/98 [00:04<00:00, 21.61it/s]

Running superstar on 49 runs...
Running superstar on CrabNebula in parallel with 12 cores.
100%|██████████| 7/7 [00:00<00:00, 32.11it/s]

Random Forest for ST0316 za35to50 weak moon found: /jgreen/MAGICPY_ANALYSIS/RFs/
Random Forest for ST0316 za35to50 strong moon found: /jgreen/MAGICPY_ANALYSIS/RFs/
Random Forest for ST0316 za35to50 decent moon found: /jgreen/MAGICPY_ANALYSIS/RFs/
All RFs found. Running melibea for CrabNebula.

Making theta2 plots...
Making skymaps...
Done analyzing CrabNebula ✨
source.go() logs   --->

(it works)

No direct contact with MARS/command line

Other features

  • automatic fast analysis
  • Dark and moon analysis
  • Easily search database
  • Download and organize data
  • Filter data by any parameter
  • Read and scan runbooks
  • Easily create RFs
  • Search and filter OFF data
  • Download MC data
  • Automatically avoid redownloading or reprocessing data
  • Easily apply moon cleanings
  • Run MARS programs in parallel
  • Run single MARS executables from python
  • Edit/change runcards/CLI params from within python
  • Lightcurves, spectra, and theta2
  • Save analyses and resume later
  • Add noise to dark OFF data for moon analysis

already implemented

In the works for v1.0

  • Add parallelization everywhere possible
  • Finish implementing PIC mode 
    • doesn't download data
    • uses condor to run jobs
  • Tool to browse/search for sources
  • Fine tune auto mode for fast analyses
  • Finish mono analysis implementation
  • Validate analyses
  • Add strict typing with mypy
  • Create good documentation
  • DL3
  • Interact with SOBO often

(just a selection shown here)

Release?

  • Still a couple more major features to implement
  • Still need pipeline validation

To answer most questions:

Can magicpy do *specific analysis thing*?

 

Does magicpy take into account *random analysis detail*?

Yes.

It's actively in development so we can ensure it does everything properly.

Let's discuss!

🧙🐍

A small demo

import magicpy.analysis as analysis
import magicpy.periods as periods

source = analysis.Source(
    name="CrabNebula",  
    period=periods.ST0316
)

source.search()


"""
Total runs: 124, Total hours: 31.55

Period: ST0316, Total runs: 118, Total hours: 30.37
+----------+------------+------------+------------+------------+
| ST0316   |   za35to50 |   za05to35 |   za50to62 |   za62to70 |
|----------+------------+------------+------------+------------|
| weak     |       4.17 |      11.46 |       5.27 |       4.26 |
| moderate |       1.21 |       0.24 |       0    |       0    |
| decent   |       1.12 |       0.97 |       0.09 |       0    |
| strong   |       0.03 |       0    |       0    |       0    |
| high     |       0    |       1.55 |       0    |       0    |
+----------+------------+------------+------------+------------+
"""

Starting an analysis

A small demo

Download and view runbooks

import datetime as dt

source.download_runbooks()


# get runbook text
source.get_runbook(
  date=dt.date(2021, 3, 16)
)


# scans runbook for your MOLA results
source.runbook_report()

"""
Scanning for CrabNebula:

------------------------------ opening runbook for 2021-02-03 ------------------------------

MOLA results:

t_eff: 6450.9 s
LE: 14.7 sigma
HE: 37.1 sigma
HE flux: 1.0 C.U.
----------------------------------- done with 2021-02-03 -----------------------------------


------------------------------ opening runbook for 2021-03-14 ------------------------------

Runbook does not follow standard setup, or source CrabNebula is somehow written differently.
----------------------------------- done with 2021-03-14 -----------------------------------
"""

A small demo

Filtering data with pandas / downloading

import magicpy.moons as moons
import magicpy.zeniths as zeniths

source.filter(
    {
        "za": [zeniths.za35to50],
        "moon_level": [moons.WeakMoon, moons.ModerateMoon, moons.DecentMoon],
        "L1_rate": ">0.0",
        "sum_trigger": "No_Sumt"
    }
)


# download data
source.get_data()


"""
Downloading data at level SuperStar to /remote/.../ALL/SuperStar
Downloading 11 files from 3 directories.
100%|██████████| 11/11 [00:00<00:00, 9067.87it/s]


Downloading data at level Calibrated to /remote/.../ALL/Calibrated
Downloading 222 files from 6 directories.
100%|██████████| 222/222 [00:00<00:00, 58137.83it/s]
"""

A small demo

moon data: apply proper cleaning levels

  • calculates rms moon levels
  • knows what cleaning levels to use per run
  • on a run-by-run bases
source.run_star()
source.run_superstar()

# 100%|██████████| 98/98 [00:25:38<00:00, 3.86s/it]

# or 

source.run_star(parallel=True)
source.run_superstar(parallel=True)

# apply random forest
source.apply_rfs()

"""
Random Forest for ST0316 za35to50 weak moon found: 
		/remote/.../RFs/ST0316/za35to50/weak

Running melibea for CrabNebula, ST0316, za35to50 and weak moon.
"""

A small demo

Train RFs in a few lines of code

import magicpy.analysis as analysis
import magicpy.periods as periods
import magicpy.moons as moons
import magicpy.zeniths as zeniths

PERIOD = periods.ST0316
ZENITH = zeniths.za35to50
MOON = moons.DecentMoon


off = analysis.Off(
    period=PERIOD, 
    zenith=ZENITH, 
    moon_level=MOON,
)


off.search()

# check combined MOLA results
# for OFF source ideas
off.get_mola()

A small demo

Train RFs in a few lines of code

Data selected :
	PKS2345-16            2 runs       0.6 h
	GB6-J0316+09          2 runs       0.5 h
	OriTrapezium          9 runs       2.9 h
	HB9SNR-R2             5 runs       1.7 h
	HydraA                2 runs       0.6 h
	GRB210321A            4 runs       1.3 h
	GRB210619             1 runs       0.1 h
	SGR1900+14            6 runs       1.8 h
	TXS0149+710           3 runs       0.5 h
	Total                34 runs      10.0 h

A small demo

Train RFs in a few lines of code

# create an MC instance as well

mc = analysis.MC(
  off_source=off
)

mc.get_data()
mc.run_star()
mc.run_superstar()


# train RFs
 
off.train(mc_data=mc)

# apply RFs to MCs

mc.apply_rfs()

magicpy (PadovvaF2F)

By astrojarred

magicpy (PadovvaF2F)

  • 176

More from astrojarred