[mlpack] Ready to use Models in mlpack (GSOC '21)

Sun Mar 28 11:48:23 EDT 2021

Slight changes to the previous email I would like to mention in my proposal
that the aim will be to try to extract the weights from PyTorch and make
the models pretrained and not just predefined using the converter but I am
not 100% sure if it would work so will that be something acceptable?

On Sun, Mar 28, 2021 at 8:44 PM Aakash kaushik <kaushikaakash7539 at gmail.com>
wrote:

> Adding to the previous email it should be easily possible to add both the
> version of deeplab_v3 which are with the resnet 50 and resnet 101 backbone
> because they have the same blocks but just different params with the
> different number of layers so I want to discuss the points mentioned in the
> previous mail but also to add to that I want to discuss the scope of the
> project should I keep it concretely to a data loader and to predefined but
> not pre-trained models which are deeplab_v3 with resnet 50 and resnet 101
> backbone.
> For an iteration, I would like to propose adding the following to just
> make it clear.
> 1. segmentation data loader
> 2. deeplabv3 (resnet50 and resnet101) predefined and not pre-trained
>
> I know we previously talked about adding another model if time permits,
> but I am not really sure about that, so I wanted to discuss these and see
> if I should add the third model to my proposal and mention the time
> constraint and just let it be a potential model or if I should completely
> remove the idea of the third model.
>
> Best,
> Aakash
>
> On Sun, Mar 28, 2021 at 2:33 PM Aakash kaushik <
> kaushikaakash7539 at gmail.com> wrote:
>
>> Hey Everyone,
>>
>> This is a continued mail regarding my proposal and I have been taking a
>> deeper look and found out that PyTorch implements these segmentations
>> models above their pre-existing pipelines of backbones and I proposed to
>> implement 3 things 1 data loader for segmentation task and 2 models from
>> which one would be a potential model only if time permits but the way I see
>> for the present time they can be implemented in mlpack is by creating block
>> and creating completed models because no such customizable backbones exist
>> as of now that can be called. and so this opens another question as to such
>> deeplab_v3 consists of three backbones in PyTorch which are resnet 50,
>> resnet 101, and mobilenet v3 large and as coco is such a huge dataset and
>> the tool for converting weights from torch to mlpack is a bit flack should
>> I go with a proposal to include predefined models which can be trained
>> rather than pre-trained models for now and if it permits we can add the
>> weights for coco in the future. I think these were some of the doubts I had
>> while I was writing the exacts of my proposal and I will be glad to have a
>> discussion on it in terms of how concrete it is and if you guys see a
>> problem on how they can be implemented or any other doubts or questions.
>>
>> Best,
>> Aakash
>>
>> On Thu, Mar 18, 2021 at 5:35 AM Aakash kaushik <
>> kaushikaakash7539 at gmail.com> wrote:
>>
>>> Hey Marcus,
>>>
>>> I totally got it and i think 1 data loader and 2 models from which 1
>>> will be a potential model if only time permits.
>>>
>>> Thank you for the feedback and help. :D
>>>
>>> Best,
>>> Aakash
>>>
>>>
>>> On Wed, 17 Mar, 2021, 9:33 pm Marcus Edel, <marcus.edel at fu-berlin.de>
>>> wrote:
>>>
>>>> Yes, that’s what I had in mine, but at the end it’s your decision.
>>>> About the
>>>> model either is fine, you can select whatever you find more interesting.
>>>>
>>>> On 17. Mar 2021, at 10:45, Aakash kaushik <kaushikaakash7539 at gmail.com>
>>>> wrote:
>>>>
>>>> HI Marcus,
>>>>
>>>> Thank you so much for reaching back, So just to clarify i would keep
>>>> the deliverables to just two which will be:
>>>>
>>>> 1. Semantic segmentation dataloader in the format of COCO dataset .
>>>> 2. One semantic segmentation model
>>>>
>>>> If I understood you correctly, will you be able to help me decide which
>>>> kind of model I should add, should i go for a model that is more generally
>>>> used such as U-Net or one from the above list that PyTorch has ?
>>>>
>>>> Best,
>>>> Aakash
>>>>
>>>> On Wed, Mar 17, 2021 at 7:55 PM Marcus Edel <marcus.edel at fu-berlin.de>
>>>> wrote:
>>>>
>>>>> Hello Aakash,
>>>>>
>>>>> thanks for the interest in the project and all the contributions; what
>>>>> you proposed
>>>>> looks quite useful to me and as you already pointed out would
>>>>> integrate really well
>>>>> with some of the existing functionalities.
>>>>>
>>>>> I guess for loading segmentation datasets we will stick with a common
>>>>> format e.g.
>>>>> COCO, and add support for the data loader and potentially add support
>>>>> for other
>>>>> formats later?
>>>>>
>>>>> One remark about the scope, you might want to remove one model from
>>>>> the list, and
>>>>> add a note to the proposal something along the lines of, if there is
>>>>> time left at the end
>>>>> of the summer, I propose to work on z, but the focus is on x and y.
>>>>>
>>>>> I hope what I said was useful; please don't hesitate to ask if
>>>>> anything needs clarification.
>>>>>
>>>>> Thanks,
>>>>> Marcus
>>>>>
>>>>> On 16. Mar 2021, at 00:16, Aakash kaushik <kaushikaakash7539 at gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hey everyone,
>>>>>
>>>>> My name is Aakash Kaushik <https://github.com/Aakash-kaushik> and I
>>>>> have been contributing for some time specifically on the ANN codebase in
>>>>> mlpack.
>>>>>
>>>>> And the project idea that is ready to use Models in mlpack peaks my
>>>>> interest. So initially i would like to propose a data loader and 2 models
>>>>> for semantic segmentation because i see that the data loaders for image
>>>>> classification and object detection are already there and including a
>>>>> couple of models and a data loader in GSOC for semantic segmentation will
>>>>> open the gates for further contribution of models in all three fields as
>>>>> they would only need to worry about the model and not loading the data and
>>>>> also will have some reference models in that field
>>>>>
>>>>> So the data loader would be capable of taking up image segmentation
>>>>> data that is the real image, segmentation map, segmentation map to class
>>>>> mapping, and for the models i am a bit confused as if we want some basic
>>>>> nets such as U-nets or a combination of both a basic net and state of the
>>>>> art model, or two state of the art model. Pytorch supports couple of models
>>>>> in the semantic segmentation fields which are:
>>>>>
>>>>> 1. FCN ResNet50, ResNet101
>>>>> 2. DeepLabV3 ResNet50, ResNet101, MobileNetV3-Large
>>>>> 3. LR-ASPP MobileNetV3-Large
>>>>>
>>>>> And so i should be able to convert their weights from pytorch to
>>>>> mlpack by modifying the utility created by kartik dutt which is
>>>>> mlpack-PyTorch-Weight-Translator
>>>>> <https://github.com/kartikdutt18/mlpack-PyTorch-Weight-Translator>
>>>>>
>>>>> I am trying to keep the deliverables to just three which is a data
>>>>> loader and 2 models as the GSOC period is reduced to just 1.5 months and
>>>>> for these three things i would have to write tests, documentation and
>>>>> example usage in the example repository.
>>>>>
>>>>> And before this work as we are in the process of removing boost
>>>>> visitors from the ANN codebase and had couple of major changes to the
>>>>> mlpack codebase the models repo wasn't able to keep up with it so my main
>>>>> goal before GSOC starts would be to work on the PR that is to  Swap
>>>>> boost::variant with vtable
>>>>> <https://github.com/mlpack/mlpack/pull/2777> and then make changes to
>>>>> the code in models repo to adjust the change in boost
>>>>> visitors, serialization and porting tests to catch2.
>>>>>
>>>>> I wanted to hear from you if this is the right path and if the number
>>>>> of deliverables are right for this and help in choosing the exact models
>>>>> that i should pick that would be the most helpful or beneficial to the
>>>>> library.
>>>>>
>>>>> Best,
>>>>> Aakash
>>>>> _______________________________________________
>>>>> mlpack mailing list
>>>>> mlpack at lists.mlpack.org
>>>>> http://knife.lugatgt.org/cgi-bin/mailman/listinfo/mlpack
>>>>>
>>>>>
>>>>>
>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://knife.lugatgt.org/pipermail/mlpack/attachments/20210328/6cf136ba/attachment-0001.htm>